#  CS109A/STAT121A/AC209A/CSCIE109A  Introduction to Data Science

## Lab 3: plotting, K-NN Regression, Simple Linear Regression

**Harvard University**

**Fall 2019**

**Instructors:** Pavlos Protopapas, Kevin Rader, and Chris Tanner

**Material prepared by**: David Sondak, Will Claybaugh, Pavlos Protopapas, and Eleni Kaxiras.

---

```python
#RUN THIS CELL
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2018-CS109A/master/content/styles/cs109.css").text
HTML(styles)
```

## Learning Goals

By the end of this lab, you should be able to:
* Review `numpy` including 2-D arrays and understand array reshaping
* Use `matplotlib` to make plots
* Feel comfortable with simple linear regression
* Feel comfortable with $k$ nearest neighbors

**This lab corresponds to lectures 4 and 5 and maps on to homework 2 and beyond.**

## Table of Contents

####  HIGHLIGHTS FROM PRE-LAB

* [1 - Review of numpy](#first-bullet)
* [2 - Intro to matplotlib plus more ](#second-bullet)

####  LAB 3 MATERIAL

* [3 - Simple Linear Regression](#third-bullet)
* [4 - Building a model with `statsmodels` and `sklearn`](#fourth-bullet)
* [5 - Example: Simple linear regression with automobile data](#fifth-bullet)
* [6 - $k$Nearest Neighbors](#sixth-bullet)

```python
import numpy as np
import scipy as sp
import matplotlib as mpl
import matplotlib.cm as cm
import matplotlib.pyplot as plt
import pandas as pd
import time
pd.set_option('display.width', 500)
pd.set_option('display.max_columns', 100)
pd.set_option('display.notebook_repr_html', True)
#import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
# Displays the plots for us.
%matplotlib inline
```

```python
# Use this as a variable to load solutions: %load PATHTOSOLUTIONS/exercise1.py. It will be substituted in the code
# so do not worry if it disappears after you run the cell.
PATHTOSOLUTIONS = 'solutions'
```

## 1 - Review of  the  `numpy` Python library

In lab1 we learned about the `numpy` library [(documentation)](http://www.numpy.org/) and its fast array structure, called the `numpy array`.

```python
# import numpy
import numpy as np
```

```python
# make an array
my_array = np.array([1,4,9,16])
my_array
```

    array([ 1,  4,  9, 16])

```python
print(f'Size of my array: {my_array.size}, or length of my array: {len(my_array)}')
print (f'Shape of my array: {my_array.shape}')
```

    Size of my array: 4, or length of my array: 4
    Shape of my array: (4,)

#### Notice the way the shape appears in numpy arrays

- For a 1D array, .shape returns a tuple with 1 element (n,)
- For a 2D array, .shape returns a tuple with 2 elements (n,m)
- For a 3D array, .shape returns a tuple with 3 elements (n,m,p)

```python
# How to reshape a 1D array to a 2D
my_array.reshape(2,-1)
```

    array([[ 1,  4],
           [ 9, 16]])

Numpy arrays support the same operations as lists! Below we slice and iterate.

```python
print("array[2:4]:", my_array[2:4]) # A slice of the array

# Iterate over the array
for ele in my_array:
    print("element:", ele)
```

    array[2:4]: [ 9 16]
    element: 1
    element: 4
    element: 9
    element: 16

Remember `numpy` gains a lot of its efficiency from being **strongly typed** (all elements are of the same type, such as integer or floating point). If the elements of an array are of a different type, `numpy` will force them into the same type (the longest in terms of bytes)

```python
mixed = np.array([1, 2.3, 'eleni', True])
print(type(1), type(2.3), type('eleni'), type(True))
mixed # all elements will become strings
```

    <class 'int'> <class 'float'> <class 'str'> <class 'bool'>

    array(['1', '2.3', 'eleni', 'True'], dtype='<U32')

Next, we push ahead to two-dimensional arrays and begin to dive into some of the deeper aspects of `numpy`.

```python
# create a 2d-array by handing a list of lists
my_array2d = np.array([ [1, 2, 3, 4],
                        [5, 6, 7, 8],
                        [9, 10, 11, 12]
])

my_array2d
```

    array([[ 1,  2,  3,  4],
           [ 5,  6,  7,  8],
           [ 9, 10, 11, 12]])

### Array Slicing (a reminder...)

Numpy arrays can be sliced, and can be iterated over with loops.  Below is a schematic illustrating slicing two-dimensional arrays.

Notice that the list slicing syntax still works!
`array[2:,3]` says "in the array, get rows 2 through the end, column 3]"
`array[3,:]` says "in the array, get row 3, all columns".

### Pandas Slicing (a reminder...)

`.iloc` is by position (position is unique), `.loc` is by label (label is not unique)

```python
# import cast dataframe
cast = pd.read_csv('../data/cast.csv', encoding='utf_8')
cast.head()
```

      title
      year
      name
      type
      character
      n

      0
      Closet Monster
      2015
      Buffy #1
      actor
      Buffy 4
      31.0

      1
      Suuri illusioni
      1985
      Homo $
      actor
      Guests
      22.0

      2
      Battle of the Sexes
      2017
      $hutter
      actor
      Bobby Riggs Fan
      10.0

      3
      Secret in Their Eyes
      2015
      $hutter
      actor
      2002 Dodger Fan
      NaN

      4
      Steve Jobs
      2015
      $hutter
      actor
      1988 Opera House Patron
      NaN

```python
# get me rows 10 to 13 (python slicing style : exclusive of end)
cast.iloc[10:13]
```

      title
      year
      name
      type
      character
      n

      10
      When the Man Went South
      2014
      Taipaleti 'Atu'ake
      actor
      Two Palms - Ua'i Paame
      8.0

      11
      Little Angel (Angelita)
      2015
      Michael 'babeepower' Viera
      actor
      Chico
      9.0

      12
      Mixing Nia
      1998
      Michael 'babeepower' Viera
      actor
      Rapper
      NaN

```python
# get me columns 0 to 2 but all rows - use head()
cast.iloc[:, 0:2].head()
```

      title
      year

      0
      Closet Monster
      2015

      1
      Suuri illusioni
      1985

      2
      Battle of the Sexes
      2017

      3
      Secret in Their Eyes
      2015

      4
      Steve Jobs
      2015

```python
# get me rows 10 to 13 AND only columns 0 to 2
cast.iloc[10:13, 0:2]
```

      title
      year

      10
      When the Man Went South
      2014

      11
      Little Angel (Angelita)
      2015

      12
      Mixing Nia
      1998

```python
# COMPARE: get me rows 10 to 13 (pandas slicing style : inclusive of end)
cast.loc[10:13]
```

      title
      year
      name
      type
      character
      n

      10
      When the Man Went South
      2014
      Taipaleti 'Atu'ake
      actor
      Two Palms - Ua'i Paame
      8.0

      11
      Little Angel (Angelita)
      2015
      Michael 'babeepower' Viera
      actor
      Chico
      9.0

      12
      Mixing Nia
      1998
      Michael 'babeepower' Viera
      actor
      Rapper
      NaN

      13
      The Replacements
      2000
      Steven 'Bear'Boyd
      actor
      Defensive Tackle - Washington Sentinels
      NaN

```python
# give me columns 'year' and 'type' by label but only for rows 5 to 10
cast.loc[5:10,['year','type']]
```

      year
      type

      5
      2015
      actor

      6
      2015
      actor

      7
      2009
      actor

      8
      2014
      actor

      9
      2014
      actor

      10
      2014
      actor

```python

```

## 2 - Plotting with matplotlib and beyond

`matplotlib` is a very powerful `python` library for making scientific plots.

We will not focus too much on the internal aspects of `matplotlib` in today's lab. There are many excellent tutorials out there for `matplotlib`.  For example,
* [`matplotlib` homepage](https://matplotlib.org/)
* [`matplotlib` tutorial](https://github.com/matplotlib/AnatomyOfMatplotlib)

Conveying your findings convincingly is an absolutely crucial part of any analysis. Therefore, you must be able to write well and make compelling visuals.  Creating informative visuals is an involved process and we won't cover that in this lab.  However, part of creating informative data visualizations means generating *readable* figures.  If people can't read your figures or have a difficult time interpreting them, they won't understand the results of your work.  Here are some non-negotiable commandments for any plot:
* Label $x$ and $y$ axes
* Axes labels should be informative
* Axes labels should be large enough to read
* Make tick labels large enough
* Include a legend if necessary
* Include a title if necessary
* Use appropriate line widths
* Use different line styles for different lines on the plot
* Use different markers for different lines

There are other important elements, but that list should get you started on your way.

We will work with `matplotlib` and `seaborn` for plotting in this class.  `matplotlib` is a very powerful `python` library for making scientific plots.  `seaborn` is a little more specialized in that it was developed for statistical data visualization.  We will cover some `seaborn` later in class. In the meantime you can look at the [seaborn documentation](https://seaborn.pydata.org)

First, let's generate some data.

#### Let's plot some functions

We will use the following three functions to make some plots:

* Logistic function:
  \begin{align*}
    f\left(z\right) = \dfrac{1}{1 + be^{-az}}
  \end{align*}
  where $a$ and $b$ are parameters.
* Hyperbolic tangent:
  \begin{align*}
    g\left(z\right) = b\tanh\left(az\right) + c
  \end{align*}
  where $a$, $b$, and $c$ are parameters.
* Rectified Linear Unit:
  \begin{align*}
    h\left(z\right) =
    \left\{
      \begin{array}{lr}
        z, \quad z > 0 \\
        \epsilon z, \quad z\leq 0
      \end{array}
    \right.
  \end{align*}
  where $\epsilon < 0$ is a small, positive parameter.

You are given the code for the first two functions.  Notice that $z$ is passed in as a `numpy` array and that the functions are returned as `numpy` arrays.  Parameters are passed in as floats.

You should write a function to compute the rectified linear unit.  The input should be a `numpy` array for $z$ and a positive float for $\epsilon$.

```python
import numpy as np

def logistic(z: np.ndarray, a: float, b: float) -> np.ndarray:
    """ Compute logistic function
      Inputs:
         a: exponential parameter
         b: exponential prefactor
         z: numpy array; domain
      Outputs:
         f: numpy array of floats, logistic function
    """

    den = 1.0 + b * np.exp(-a * z)
    return 1.0 / den

def stretch_tanh(z: np.ndarray, a: float, b: float, c: float) -> np.ndarray:
    """ Compute stretched hyperbolic tangent
      Inputs:
         a: horizontal stretch parameter (a>1 implies a horizontal squish)
         b: vertical stretch parameter
         c: vertical shift parameter
         z: numpy array; domain
      Outputs:
         g: numpy array of floats, stretched tanh
    """
    return b * np.tanh(a * z) + c

def relu(z: np.ndarray, eps: float = 0.01) -> np.ndarray:
    """ Compute rectificed linear unit
      Inputs:
         eps: small positive parameter
         z: numpy array; domain
      Outputs:
         h: numpy array; relu
    """
    return np.fmax(z, eps * z)
```

Now let's make some plots.  First, let's just warm up and plot the logistic function.

```python
x = np.linspace(-5.0, 5.0, 100) # Equally spaced grid of 100 pts between -5 and 5

f = logistic(x, 1.0, 1.0) # Generate data
```

```python
plt.plot(x, f)
plt.xlabel('x')
plt.ylabel('f')
plt.title('Logistic Function')
plt.grid(True)
```

#### Figures with subplots

Let's start thinking about the plots as objects. We have the `figure` object which is like a matrix of smaller plots named `axes`. You can use array notation when handling it.

```python
fig, ax = plt.subplots(1,1) # Get figure and axes objects

ax.plot(x, f) # Make a plot

# Create some labels
ax.set_xlabel('x')
ax.set_ylabel('f')
ax.set_title('Logistic Function')

# Grid
ax.grid(True)
```

Wow, it's *exactly* the same plot!  Notice, however, the use of `ax.set_xlabel()` instead of `plt.xlabel()`.  The difference is tiny, but you should be aware of it.  I will use this plotting syntax from now on.

What else do we need to do to make this figure better?  Here are some options:
* Make labels bigger!
* Make line fatter
* Make tick mark labels bigger
* Make the grid less pronounced
* Make figure bigger

Let's get to it.

```python
fig, ax = plt.subplots(1,1, figsize=(10,6)) # Make figure bigger

# Make line plot
ax.plot(x, f, lw=4)

# Update ticklabel size
ax.tick_params(labelsize=24)

# Make labels
ax.set_xlabel(r'$x$', fontsize=24) # Use TeX for mathematical rendering
ax.set_ylabel(r'$f(x)$', fontsize=24) # Use TeX for mathematical rendering
ax.set_title('Logistic Function', fontsize=24)

ax.grid(True, lw=1.5, ls='--', alpha=0.75)
```

Notice:
* `lw` stands for `linewidth`.  We could also write `ax.plot(x, f, linewidth=4)`
* `ls` stands for `linestyle`.
* `alpha` stands for transparency.

The only thing remaining to do is to change the $x$ limits.  Clearly these should go from $-5$ to $5$.

```python
#fig.savefig('logistic.png')

# Put this in a markdown cell and uncomment this to check what you saved.
#
```

#### Resources
If you want to see all the styles available, please take a look at the documentation.
* [Line styles](https://matplotlib.org/2.0.1/api/lines_api.html#matplotlib.lines.Line2D.set_linestyle)
* [Marker styles](https://matplotlib.org/2.0.1/api/markers_api.html#module-matplotlib.markers)
* [Everything you could ever want](https://matplotlib.org/2.0.1/api/lines_api.html#matplotlib.lines.Line2D.set_marker)

We haven't discussed it yet, but you can also put a legend on a figure.  You'll do that in the next exercise.  Here are some additional resources:
* [Legend](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.legend.html)
* [Grid](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.grid.html)

`ax.legend(loc='best', fontsize=24);`

Exercise

Do the following:
* Make a figure with the logistic function, hyperbolic tangent, and rectified linear unit.
* Use different line styles for each plot
* Put a legend on your figure

Here's an example of a figure:

```python
# your code here

# First get the data
f = logistic(x, 2.0, 1.0)
g = stretch_tanh(x, 2.0, 0.5, 0.5)
h = relu(x)

fig, ax = plt.subplots(1,1, figsize=(10,6)) # Create figure object

# Make actual plots
# (Notice the label argument!)
ax.plot(x, f, lw=4, ls='-', label=r'$L(x;1)$')
ax.plot(x, g, lw=4, ls='--', label=r'$\tanh(2x)$')
ax.plot(x, h, lw=4, ls='-.', label=r'$relu(x; 0.01)$')

# Make the tick labels readable
ax.tick_params(labelsize=24)

# Set axes limits to make the scale nice
ax.set_xlim(x.min(), x.max())
ax.set_ylim(h.min(), 1.1)

# Make readable labels
ax.set_xlabel(r'$x$', fontsize=24)
ax.set_ylabel(r'$h(x)$', fontsize=24)
ax.set_title('Activation Functions', fontsize=24)

# Set up grid
ax.grid(True, lw=1.75, ls='--', alpha=0.75)

# Put legend on figure
ax.legend(loc='best', fontsize=24);

fig.savefig('../images/nice_plots.png')
```

Exercise

These figures look nice in the plot and it makes sense for comparison. Now let's put the 3 different figures in separate plots.

* Make a separate plot for each figure and line them up on the same row.

```python
# your code here

```

```python
# %load solutions/three_subplots.py
```

Exercise

* Make a grid of 2 x 3 separate plots, 3 will be empty. Just plot the functions and do not worry about cosmetics. We just want you ro see the functionality.

```python
# your code here

```

```python
# %load solutions/six_subplots.py
```

## 3 - Simple Linear Regression

Linear regression and its many extensions are a workhorse of the statistics and data science community, both in application and as a reference point for other models. Most of the major concepts in machine learning can be and often are discussed in terms of various linear regression models. Thus, this section will introduce you to building and fitting linear regression models and some of the process behind it, so that you can 1) fit models to data you encounter 2) experiment with different kinds of linear regression and observe their effects 3) see some of the technology that makes regression models work.

### Linear regression with a toy dataset
We first examine a toy problem, focusing our efforts on fitting a linear model to a small dataset with three observations.  Each observation consists of one predictor $x_i$ and one response $y_i$ for $i = 1, 2, 3$,

\begin{align*}
(x , y) = \{(x_1, y_1), (x_2, y_2), (x_3, y_3)\}.
\end{align*}

To be very concrete, let's set the values of the predictors and responses.

\begin{equation*}
(x , y) = \{(1, 2), (2, 2), (3, 4)\}
\end{equation*}

There is no line of the form $\beta_0 + \beta_1 x = y$ that passes through all three observations, since the data are not collinear. Thus our aim is to find the line that best fits these observations in the *least-squares sense*, as discussed in lecture.

Exercise (for home)

* Make two numpy arrays out of this data, x_train and y_train
* Check the dimentions of these arrays
* Try to reshape them into a different shape
* Make points into a very simple scatterplot
* Make a better scatterplot

```python
# your code here
```

```python
# solution
x_train = np.array([1,2,3])
y_train = np.array([2,3,6])
type(x_train)
```

```python
x_train.shape
```

```python
x_train = x_train.reshape(3,1)
x_train.shape
```

```python
xx = np.array([[1,3,5],[6,2,1]])
xx.shape
```

```python
xx = xx.reshape(3,-1)
xx
```

```python
# %load solutions/simple_scatterplot.py
# Make a simple scatterplot
plt.scatter(x_train,y_train)

# check dimensions
print(x_train.shape,y_train.shape)

```

```python
# %load solutions/nice_scatterplot.py
def nice_scatterplot(x, y, title):
    # font size
    f_size = 18

    # make the figure
    fig, ax = plt.subplots(1,1, figsize=(8,5)) # Create figure object

    # set axes limits to make the scale nice
    ax.set_xlim(np.min(x)-1, np.max(x) + 1)
    ax.set_ylim(np.min(y)-1, np.max(y) + 1)

    # adjust size of tickmarks in axes
    ax.tick_params(labelsize = f_size)

    # remove tick labels
    ax.tick_params(labelbottom=False,  bottom=False)

    # adjust size of axis label
    ax.set_xlabel(r'$x$', fontsize = f_size)
    ax.set_ylabel(r'$y$', fontsize = f_size)

    # set figure title label
    ax.set_title(title, fontsize = f_size)

    # you may set up grid with this
    ax.grid(True, lw=1.75, ls='--', alpha=0.15)

    # make actual plot (Notice the label argument!)
    #ax.scatter(x, y, label=r'$my points$')
    #ax.scatter(x, y, label='$my points$')
    ax.scatter(x, y, label=r'$my\,points$')
    ax.legend(loc='best', fontsize = f_size);

    return ax

nice_scatterplot(x_train, y_train, 'hello nice plot')

```

#### Formulae
Linear regression is special among the models we study because it can be solved explicitly. While most other models (and even some advanced versions of linear regression) must be solved itteratively, linear regression has a formula where you can simply plug in the data.

For the single predictor case it is:
    \begin{align}
      \beta_1 &= \frac{\sum_{i=1}^n{(x_i-\bar{x})(y_i-\bar{y})}}{\sum_{i=1}^n{(x_i-\bar{x})^2}}\\
      \beta_0 &= \bar{y} - \beta_1\bar{x}\
    \end{align}

Where $\bar{y}$ and $\bar{x}$ are the mean of the y values and the mean of the x values, respectively.

From the re-aranged second equation we can see that the best-fit line  passes through $(\bar{x},\bar{y})$, the center of mass of the data

From any of the first equations, we can see that the slope of the line has to do with whether or not an x value that is above/below the center of mass is typically paired with a y value that is likewise above/below, or typically paired with one that is opposite.

###  Building a model from scratch
In this part, we will solve the equations for simple linear regression and find the best fit solution to our toy problem.

The snippets of code below implement the linear regression equations on the observed predictors and responses, which we'll call the training data set.  Let's walk through the code.

We have to reshape our arrrays to 2D. We will see later why.

Exercise

* make an array with shape (2,3)
* reshape it to a size that you want

```python
# your code here
xx = np.array([[1,2,3],[4,6,8]])
xxx = xx.reshape(-1,2)
xxx.shape
```

```python
# Reshape to be a proper 2D array
x_train = x_train.reshape(x_train.shape[0], 1)
y_train = y_train.reshape(y_train.shape[0], 1)

print(x_train.shape)
```

```python
# first, compute means
y_bar = np.mean(y_train)
x_bar = np.mean(x_train)

# build the two terms
numerator = np.sum( (x_train - x_bar)*(y_train - y_bar) )
denominator = np.sum((x_train - x_bar)**2)

print(numerator.shape, denominator.shape) #check shapes
```

* Why the empty brackets? (The numerator and denominator are scalars, as expected.)

```python
#slope beta1
beta_1 = numerator/denominator

#intercept beta0
beta_0 = y_bar - beta_1*x_bar

print("The best-fit line is {0:3.2f} + {1:3.2f} * x".format(beta_0, beta_1))
print(f'The best fit is {beta_0}')
```

Exercise

Turn the code from the above cells into a function called `simple_linear_regression_fit`, that inputs the training data and returns `beta0` and `beta1`.

To do this, copy and paste the code from the above cells below and adjust the code as needed, so that the training data becomes the input and the betas become the output.

```python
def simple_linear_regression_fit(x_train: np.ndarray, y_train: np.ndarray) -> np.ndarray:

    return
```

Check your function by calling it with the training data from above and printing out the beta values.

```python
# Your code here
```

```python
# %load solutions/simple_linear_regression_fit.py
def simple_linear_regression_fit(x_train: np.ndarray, y_train: np.ndarray) -> np.ndarray:
    """
    Inputs:
    x_train: a (num observations by 1) array holding the values of the predictor variable
    y_train: a (num observations by 1) array holding the values of the response variable

    Returns:
    beta_vals:  a (num_features by 1) array holding the intercept and slope coeficients
    """

    # Check input array sizes
    if len(x_train.shape) < 2:
        print("Reshaping features array.")
        x_train = x_train.reshape(x_train.shape[0], 1)

    if len(y_train.shape) < 2:
        print("Reshaping observations array.")
        y_train = y_train.reshape(y_train.shape[0], 1)

    # first, compute means
    y_bar = np.mean(y_train)
    x_bar = np.mean(x_train)

    # build the two terms
    numerator = np.sum( (x_train - x_bar)*(y_train - y_bar) )
    denominator = np.sum((x_train - x_bar)**2)

    #slope beta1
    beta_1 = numerator/denominator

    #intercept beta0
    beta_0 = y_bar - beta_1*x_bar

    return np.array([beta_0,beta_1])

```

* Let's run this function and see the coefficients

```python
x_train = np.array([1 ,2, 3])
y_train = np.array([2, 2, 4])

betas = simple_linear_regression_fit(x_train, y_train)

beta_0 = betas[0]
beta_1 = betas[1]

print("The best-fit line is {0:8.6f} + {1:8.6f} * x".format(beta_0, beta_1))
```

Exercise

* Do the values of `beta0` and `beta1` seem reasonable?
* Plot the training data using a scatter plot.
* Plot the best fit line with `beta0` and `beta1` together with the training data.

```python
# Your code here
```

```python
# %load solutions/best_fit_scatterplot.py
fig_scat, ax_scat = plt.subplots(1,1, figsize=(10,6))

# Plot best-fit line
x_train = np.array([[1, 2, 3]]).T

best_fit = beta_0 + beta_1 * x_train

ax_scat.scatter(x_train, y_train, s=300, label='Training Data')
ax_scat.plot(x_train, best_fit, ls='--', label='Best Fit Line')

ax_scat.set_xlabel(r'$x_{train}$')
ax_scat.set_ylabel(r'$y$');

```

The values of `beta0` and `beta1` seem roughly reasonable.  They capture the positive correlation.  The line does appear to be trying to get as close as possible to all the points.

## 4 - Building a model with `statsmodels` and `sklearn`

Now that we can concretely fit the training data from scratch, let's learn two `python` packages to do it all for us:
* [statsmodels](http://www.statsmodels.org/stable/regression.html) and
* [scikit-learn (sklearn)](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html).

Our goal  is to show how to implement simple linear regression with these packages.  For an important sanity check, we compare the $\beta$ values from `statsmodels` and `sklearn` to the $\beta$ values that we found from above with our own implementation.

For the purposes of this lab, `statsmodels` and `sklearn` do the same thing.  More generally though, `statsmodels` tends to be easier for inference \[finding the values of the slope and intercept and dicussing uncertainty in those values\], whereas `sklearn` has machine-learning algorithms and is better for prediction \[guessing y values for a given x value\]. (Note that both packages make the same guesses, it's just a question of which activity they provide more support for.

**Note:** `statsmodels` and `sklearn` are different packages!  Unless we specify otherwise, you can use either one.

Below is the code for `statsmodels`.  `Statsmodels` does not by default include the column of ones in the $X$ matrix, so we include it manually with `sm.add_constant`.

```python
import statsmodels.api as sm
```

```python
# create the X matrix by appending a column of ones to x_train
X = sm.add_constant(x_train)

# this is the same matrix as in our scratch problem!
print(X)

# build the OLS model (ordinary least squares) from the training data
toyregr_sm = sm.OLS(y_train, X)

# do the fit and save regression info (parameters, etc) in results_sm
results_sm = toyregr_sm.fit()

# pull the beta parameters out from results_sm
beta0_sm = results_sm.params[0]
beta1_sm = results_sm.params[1]

print(f'The regression coef from statsmodels are: beta_0 = {beta0_sm:8.6f} and beta_1 = {beta1_sm:8.6f}')
```

Besides the beta parameters, `results_sm` contains a ton of other potentially useful information.

```python
import warnings
warnings.filterwarnings('ignore')
print(results_sm.summary())
```

Now let's turn our attention to the `sklearn` library.

```python
from sklearn import linear_model
```

```python
# build the least squares model
toyregr = linear_model.LinearRegression()

# save regression info (parameters, etc) in results_skl
results = toyregr.fit(x_train, y_train)

# pull the beta parameters out from results_skl
beta0_skl = toyregr.intercept_
beta1_skl = toyregr.coef_[0]

print("The regression coefficients from the sklearn package are: beta_0 = {0:8.6f} and beta_1 = {1:8.6f}".format(beta0_skl, beta1_skl))
```

We should feel pretty good about ourselves now, and we're ready to move on to a real problem!

### The `scikit-learn` library and the shape of things

Before diving into a "real" problem, let's discuss more of the details of `sklearn`.

`Scikit-learn` is the main `Python` machine learning library. It consists of many learners which can learn models from data, as well as a lot of utility functions such as `train_test_split()`.

Use the following to add the library into your code:

```python
import sklearn
```

In `scikit-learn`, an **estimator** is a Python object that implements the methods `fit(X, y)` and `predict(T)`

Let's see the structure of `scikit-learn` needed to make these fits. `fit()` always takes two arguments:
```python
estimator.fit(Xtrain, ytrain)
```
We will consider two estimators in this lab: `LinearRegression` and `KNeighborsRegressor`.

It is very important to understand that `Xtrain` must be in the form of a **2x2 array** with each row corresponding to one sample, and each column corresponding to the feature values for that sample.

`ytrain` on the other hand is a simple array of responses.  These are continuous for regression problems.

### Practice with `sklearn` and a real dataset
We begin by loading up the `mtcars` dataset. This data was extracted from the 1974 Motor Trend US magazine, and comprises of fuel consumption and 10 aspects of automobile design and performance for 32 automobiles (1973–74 models). We will load this data to a dataframe with 32 observations on 11 (numeric) variables. Here is an explanation of the features:

- `mpg` is Miles/(US) gallon
- `cyl` is Number of cylinders,
- `disp` is	Displacement (cu.in.),
- `hp` is	Gross horsepower,
- `drat` is	Rear axle ratio,
- `wt` is the Weight (1000 lbs),
- `qsec` is 1/4 mile time,
- `vs` is Engine (0 = V-shaped, 1 = straight),
- `am` is Transmission (0 = automatic, 1 = manual),
- `gear` is the Number of forward gears,
- `carb` is	Number of carburetors.

```python
import pandas as pd

#load mtcars
dfcars = pd.read_csv("../data/mtcars.csv")
dfcars.head()
```

```python
# Fix the column title
dfcars = dfcars.rename(columns={"Unnamed: 0":"car name"})
dfcars.head()
```

```python
dfcars.shape
```

#### Searching for values: how many cars have 4 gears?

```python
len(dfcars[dfcars.gear == 4].drop_duplicates(subset='car name', keep='first'))
```

Next, let's split the dataset into a training set and test set.

```python
# split into training set and testing set
from sklearn.model_selection import train_test_split

#set random_state to get the same split every time
traindf, testdf = train_test_split(dfcars, test_size=0.2, random_state=42)
```

```python
# testing set is around 20% of the total data; training set is around 80%
print("Shape of full dataset is: {0}".format(dfcars.shape))
print("Shape of training dataset is: {0}".format(traindf.shape))
print("Shape of test dataset is: {0}".format(testdf.shape))
```

Now we have training and test data.  We still need to select a predictor and a response from this dataset.  Keep in mind that we need to choose the predictor and response from both the training and test set.  You will do this in the exercises below.  However, we provide some starter code for you to get things going.

```python
traindf.head()
```

```python
# Extract the response variable that we're interested in
y_train = traindf.mpg
y_train
```

Exercise

Use slicing to get the same vector `y_train`

----

Now, notice the shape of `y_train`.

```python
y_train.shape, type(y_train)
```

### Array reshape
This is a 1D array as should be the case with the **Y** array.  Remember, `sklearn` requires a 2D array only for the predictor array.  You will have to pay close attention to this in the exercises later. `Sklearn` doesn't care too much about the shape of `y_train`.

The whole reason we went through that whole process was to show you how to reshape your data into the correct format.

**IMPORTANT:** Remember that your response variable `ytrain` can be a vector but your predictor variable `xtrain` ***must*** be an array!

## 5 - Example: Simple linear regression with automobile data
We will now use `sklearn` to predict automobile mileage per gallon (mpg) and evaluate these predictions. We already loaded the data and split them into a training set and a test set.

We need to choose the variables that we think will be good predictors for the dependent variable `mpg`.

Exercise in pairs

* Pick one variable to use as a predictor for simple linear regression.  Discuss your reasons with the person next to you.
* Justify your choice with some visualizations.
* Is there a second variable you'd like to use? For example, we're not doing multiple linear regression here, but if we were, is there another variable you'd like to include if we were using two predictors?

```python
x_wt = dfcars.wt
x_wt.shape
```

```python
# Your code here

```

```python
# %load solutions/cars_simple_EDA.py
```

Exercise

* Use `sklearn` to fit the training data using simple linear regression.
* Use the model to make mpg predictions on the test set.
* Plot the data and the prediction.
* Print out the mean squared error for the training set and the test set and compare.

```python
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

dfcars = pd.read_csv("../data/mtcars.csv")
dfcars = dfcars.rename(columns={"Unnamed: 0":"name"})

dfcars.head()
```

```python
traindf, testdf = train_test_split(dfcars, test_size=0.2, random_state=42)

y_train = np.array(traindf.mpg)
X_train = np.array(traindf.wt)
X_train = X_train.reshape(X_train.shape[0], 1)
```

```python
y_test = np.array(testdf.mpg)
X_test = np.array(testdf.wt)
X_test = X_test.reshape(X_test.shape[0], 1)
```

```python
# Let's take another look at our data
dfcars.head()
```

```python
# And out train and test sets
y_train.shape, X_train.shape
```

```python
y_test.shape, X_test.shape
```

```python
#create linear model
regression = LinearRegression()

#fit linear model
regression.fit(X_train, y_train)

predicted_y = regression.predict(X_test)

r2 = regression.score(X_test, y_test)
print(f'R^2 = {r2:.5}')
```

```python
print(regression.score(X_train, y_train))

print(mean_squared_error(predicted_y, y_test))
print(mean_squared_error(y_train, regression.predict(X_train)))

print('Coefficients: \n', regression.coef_[0], regression.intercept_)
```

```python
fig, ax = plt.subplots(1,1, figsize=(10,6))
ax.plot(y_test, predicted_y, 'o')
grid = np.linspace(np.min(dfcars.mpg), np.max(dfcars.mpg), 100)
ax.plot(grid, grid, color="black") # 45 degree line
ax.set_xlabel("actual y")
ax.set_ylabel("predicted y")

fig1, ax1 = plt.subplots(1,1, figsize=(10,6))
ax1.plot(dfcars.wt, dfcars.mpg, 'o')
xgrid = np.linspace(np.min(dfcars.wt), np.max(dfcars.wt), 100)
ax1.plot(xgrid, regression.predict(xgrid.reshape(100, 1)))
```

## 6 - $k$-nearest neighbors

Now that you're familiar with `sklearn`, you're ready to do a KNN regression.

Sklearn's regressor is called `sklearn.neighbors.KNeighborsRegressor`. Its main parameter is the `number of nearest neighbors`. There are other parameters such as the distance metric (default for 2 order is the Euclidean distance). For a list of all the parameters see the [Sklearn kNN Regressor Documentation](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsRegressor.html).

Let's use $5$ nearest neighbors.

```python
# Import the library
from sklearn.neighbors import KNeighborsRegressor
```

```python
# Set number of neighbors
k = 5
knnreg = KNeighborsRegressor(n_neighbors=k)
```

```python
# Fit the regressor - make sure your numpy arrays are the right shape
knnreg.fit(X_train, y_train)

# Evaluate the outcome on the train set using R^2
r2_train = knnreg.score(X_train, y_train)

# Print results
print(f'kNN model with {k} neighbors gives R^2 on the train set: {r2_train:.5}')
```

```python
knnreg.predict(X_test)
```

Exercise

Calculate and print the $R^{2}$ score on the test set

```python
# Your code here
```

Not so good? Lets vary the number of neighbors and see what we get.

```python
# Make our lives easy by storing the different regressors in a dictionary
regdict = {}

# Make our lives easier by entering the k values from a list
k_list = [1, 2, 4, 15]

# Do a bunch of KNN regressions
for k in k_list:
    knnreg = KNeighborsRegressor(n_neighbors=k)
    knnreg.fit(X_train, y_train)
    # Store the regressors in a dictionary
    regdict[k] = knnreg

# Print the dictionary to see what we have
regdict
```

Now let's plot all the k values in same plot.

```python
fig, ax = plt.subplots(1,1, figsize=(10,6))

ax.plot(dfcars.wt, dfcars.mpg, 'o', label="data")

xgrid = np.linspace(np.min(dfcars.wt), np.max(dfcars.wt), 100)

# let's unpack the dictionary to its elements (items) which is the k and Regressor
for k, regressor in regdict.items():
    predictions = regressor.predict(xgrid.reshape(-1,1))
    ax.plot(xgrid, predictions, label="{}-NN".format(k))

ax.legend();
```

Exercise

Explain what you see in the graph. **Hint** Notice how the $1$-NN goes through every point on the training set but utterly fails elsewhere.

Lets look at the scores on the training set.

```python
ks = range(1, 15) # Grid of k's
scores_train = [] # R2 scores
for k in ks:
    # Create KNN model
    knnreg = KNeighborsRegressor(n_neighbors=k)

    # Fit the model to training data
    knnreg.fit(X_train, y_train)

    # Calculate R^2 score
    score_train = knnreg.score(X_train, y_train)
    scores_train.append(score_train)

# Plot
fig, ax = plt.subplots(1,1, figsize=(12,8))
ax.plot(ks, scores_train,'o-')
ax.set_xlabel(r'$k$')
ax.set_ylabel(r'$R^{2}$')
```

Exercise

* Why do we get a perfect $R^2$ at k=1 for the training set?
* Make the same plot as above on the *test* set.
* What is the best $k$?

```python
# Your code here

```

```python
# %load solutions/knn_regression.py
```

```python
# solution to previous exercise
r2_test = knnreg.score(X_test, y_test)
print(f'kNN model with {k} neighbors gives R^2 on the test set: {r2_test:.5}')
```


# The storage

When you use pylbm, a generated code is performed using the descritpion of the scheme(s) (the velocities, the polynomials, the conserved moments, the equilibriums, ...). There are several generators already implemented

- NumPy
- Cython
- Pythran (work in progress)
- Loo.py (work in progress)

To have best performance following the generator, you need a specific storage of the moments and distribution functions arrays. For example, it is preferable to have a storage like $[n_v, n_x, n_y, n_z]$ in NumPy $n_v$ is the number of velocities and $n_x$, $n_y$ and $n_z$ the grid size. It is due to the vectorized form of the algorithm. Whereas for Cython, it is preferable to have the storage $[n_x, n_y, n_z, n_v]$ using the pull algorithm.

So, we have implemented a storage class that always gives to the user the same access to the moments and disribution functions arrays but with a different storage in memory for the generator. This class is called [Array](module/module_storage.rst).

It is really simple to create an array. You just need to give

- the number of velocities,
- the global grid size,
- the size of the fictitious point in each direction,
- the order of $[n_v, n_x, n_y, n_z]$ with the following indices
    - 0: $n_v$
    - 1: $n_x$
    - 2: $n_y$
    - 3: $n_z$

  The default order is $[n_v, n_x, n_y, n_z]$.

- the mpi topology (optional)
- the type of the data (optional)

    The default is double

## 2D example

Suppose that you want to create an array with a grid size $[5, 10]$ and $9$ velocities with $1$ cell in each direction for the fictitious domain.

```python
from pylbm.storage import Array
import numpy as np
a = Array(9, [5, 10], [1, 1])
```

```python
for i in range(a.nv):
    a[i] = i
```

```python
print(a[:])
```

    [[[ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
      [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
      [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
      [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
      [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]]

     [[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]
      [ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]
      [ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]
      [ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]
      [ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]]

     [[ 2.  2.  2.  2.  2.  2.  2.  2.  2.  2.]
      [ 2.  2.  2.  2.  2.  2.  2.  2.  2.  2.]
      [ 2.  2.  2.  2.  2.  2.  2.  2.  2.  2.]
      [ 2.  2.  2.  2.  2.  2.  2.  2.  2.  2.]
      [ 2.  2.  2.  2.  2.  2.  2.  2.  2.  2.]]

     [[ 3.  3.  3.  3.  3.  3.  3.  3.  3.  3.]
      [ 3.  3.  3.  3.  3.  3.  3.  3.  3.  3.]
      [ 3.  3.  3.  3.  3.  3.  3.  3.  3.  3.]
      [ 3.  3.  3.  3.  3.  3.  3.  3.  3.  3.]
      [ 3.  3.  3.  3.  3.  3.  3.  3.  3.  3.]]

     [[ 4.  4.  4.  4.  4.  4.  4.  4.  4.  4.]
      [ 4.  4.  4.  4.  4.  4.  4.  4.  4.  4.]
      [ 4.  4.  4.  4.  4.  4.  4.  4.  4.  4.]
      [ 4.  4.  4.  4.  4.  4.  4.  4.  4.  4.]
      [ 4.  4.  4.  4.  4.  4.  4.  4.  4.  4.]]

     [[ 5.  5.  5.  5.  5.  5.  5.  5.  5.  5.]
      [ 5.  5.  5.  5.  5.  5.  5.  5.  5.  5.]
      [ 5.  5.  5.  5.  5.  5.  5.  5.  5.  5.]
      [ 5.  5.  5.  5.  5.  5.  5.  5.  5.  5.]
      [ 5.  5.  5.  5.  5.  5.  5.  5.  5.  5.]]

     [[ 6.  6.  6.  6.  6.  6.  6.  6.  6.  6.]
      [ 6.  6.  6.  6.  6.  6.  6.  6.  6.  6.]
      [ 6.  6.  6.  6.  6.  6.  6.  6.  6.  6.]
      [ 6.  6.  6.  6.  6.  6.  6.  6.  6.  6.]
      [ 6.  6.  6.  6.  6.  6.  6.  6.  6.  6.]]

     [[ 7.  7.  7.  7.  7.  7.  7.  7.  7.  7.]
      [ 7.  7.  7.  7.  7.  7.  7.  7.  7.  7.]
      [ 7.  7.  7.  7.  7.  7.  7.  7.  7.  7.]
      [ 7.  7.  7.  7.  7.  7.  7.  7.  7.  7.]
      [ 7.  7.  7.  7.  7.  7.  7.  7.  7.  7.]]

     [[ 8.  8.  8.  8.  8.  8.  8.  8.  8.  8.]
      [ 8.  8.  8.  8.  8.  8.  8.  8.  8.  8.]
      [ 8.  8.  8.  8.  8.  8.  8.  8.  8.  8.]
      [ 8.  8.  8.  8.  8.  8.  8.  8.  8.  8.]
      [ 8.  8.  8.  8.  8.  8.  8.  8.  8.  8.]]]

```python
b = Array(9, [5, 10], [1, 1], sorder=[2, 1, 0])
for i in range(b.nv):
    b[i] = i
```

```python
print(b[:])
```

    [[[ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
      [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
      [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
      [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
      [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]]

     [[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]
      [ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]
      [ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]
      [ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]
      [ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]]

     [[ 2.  2.  2.  2.  2.  2.  2.  2.  2.  2.]
      [ 2.  2.  2.  2.  2.  2.  2.  2.  2.  2.]
      [ 2.  2.  2.  2.  2.  2.  2.  2.  2.  2.]
      [ 2.  2.  2.  2.  2.  2.  2.  2.  2.  2.]
      [ 2.  2.  2.  2.  2.  2.  2.  2.  2.  2.]]

     [[ 3.  3.  3.  3.  3.  3.  3.  3.  3.  3.]
      [ 3.  3.  3.  3.  3.  3.  3.  3.  3.  3.]
      [ 3.  3.  3.  3.  3.  3.  3.  3.  3.  3.]
      [ 3.  3.  3.  3.  3.  3.  3.  3.  3.  3.]
      [ 3.  3.  3.  3.  3.  3.  3.  3.  3.  3.]]

     [[ 4.  4.  4.  4.  4.  4.  4.  4.  4.  4.]
      [ 4.  4.  4.  4.  4.  4.  4.  4.  4.  4.]
      [ 4.  4.  4.  4.  4.  4.  4.  4.  4.  4.]
      [ 4.  4.  4.  4.  4.  4.  4.  4.  4.  4.]
      [ 4.  4.  4.  4.  4.  4.  4.  4.  4.  4.]]

     [[ 5.  5.  5.  5.  5.  5.  5.  5.  5.  5.]
      [ 5.  5.  5.  5.  5.  5.  5.  5.  5.  5.]
      [ 5.  5.  5.  5.  5.  5.  5.  5.  5.  5.]
      [ 5.  5.  5.  5.  5.  5.  5.  5.  5.  5.]
      [ 5.  5.  5.  5.  5.  5.  5.  5.  5.  5.]]

     [[ 6.  6.  6.  6.  6.  6.  6.  6.  6.  6.]
      [ 6.  6.  6.  6.  6.  6.  6.  6.  6.  6.]
      [ 6.  6.  6.  6.  6.  6.  6.  6.  6.  6.]
      [ 6.  6.  6.  6.  6.  6.  6.  6.  6.  6.]
      [ 6.  6.  6.  6.  6.  6.  6.  6.  6.  6.]]

     [[ 7.  7.  7.  7.  7.  7.  7.  7.  7.  7.]
      [ 7.  7.  7.  7.  7.  7.  7.  7.  7.  7.]
      [ 7.  7.  7.  7.  7.  7.  7.  7.  7.  7.]
      [ 7.  7.  7.  7.  7.  7.  7.  7.  7.  7.]
      [ 7.  7.  7.  7.  7.  7.  7.  7.  7.  7.]]

     [[ 8.  8.  8.  8.  8.  8.  8.  8.  8.  8.]
      [ 8.  8.  8.  8.  8.  8.  8.  8.  8.  8.]
      [ 8.  8.  8.  8.  8.  8.  8.  8.  8.  8.]
      [ 8.  8.  8.  8.  8.  8.  8.  8.  8.  8.]
      [ 8.  8.  8.  8.  8.  8.  8.  8.  8.  8.]]]

You can see that the access of the data is the same for $a$ et $b$ whereas the sorder is not the same.

If we look at the *array* attribute which is the real storage of our data

```python
a.array.shape
```

    (9, 5, 10)

```python
b.array.shape
```

    (10, 5, 9)

you can see that it is not the same and it is exactly what we want. To do that, we use the [swapaxes](http://docs.scipy.org/doc/numpy/reference/generated/numpy.swapaxes.html) of numpy and we use this representation to have an access to our data.

## Access to the data with the conserved moments

When you discribe your scheme, you define the conserved moments. It is usefull to have a direct acces to these moments by giving their name and not their indices in the array. So, it is possible to specify where are the conserved moments in the array.

Let define conserved moments using sympy symbol.

```python
import sympy
rho, u, v = sympy.symbols("rho, u, v")
```

We indicate to pylbm where are located these conserved moments in our array by giving a list of two elements: the first one is the scheme number and the second one the index in this scheme.

```python
a.set_conserved_moments({rho: [0, 0], u: [0, 2], v: [0, 1]})
```

```python
a[rho]
```

    array([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]])

```python
a[u]
```

    array([[ 2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.],
           [ 2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.],
           [ 2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.],
           [ 2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.],
           [ 2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.]])

```python
a[v]
```

    array([[ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.],
           [ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.],
           [ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.],
           [ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.],
           [ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.]])

```python

```


```python
from sympy import *
```

```python
import pylatex as p
```

$$
  w_{t+1} = (1 + r_{t+1}) s(w_t) + y_{t+1}
$$ (my_other_label)

- A link to an equation directive: {eq}`my_label`
- A link to a dollar math block: {eq}`my_other_label`

::::{important}
:::{note}
This text is **standard** _Markdown_
:::
::::

```python

```


# Continuous Control

The environment for this project is [Reacher](https://github.com/udacity/deep-reinforcement-learning/tree/master/p2_continuous_control) from Unity, and it's provided in the `setup` folder. We'll implement the A2C algorithm as the synchronous version of [A3C](https://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf) (but the input won't be directly from pixels). We'll also include a variation called Generalized Advantage Estimation, or [GAE](https://arxiv.org/pdf/1506.02438.pdf) for short. Results will be shown in this notebook and the best solution will be implemented in `main.py`.

> The model used to generate this gif is `final.pth` (A2C-GAE with `n = 8` steps), which was trained for 600 episodes using `main.py`.

## 1. Prepare dependencies and environment

Take a look at README.md before executing this notebook and make sure that the kernel is set to **p2_continuous_control**.

```python
!pip -q install ./setup

import sys
import platform

import numpy as np
from scipy import signal
import matplotlib.pyplot as plt

import torch
import torch.nn.functional as F
import torch.optim as optim
from torch.distributions.normal import Normal
from torch import nn

from setup import unityagents
from unityagents import UnityEnvironment

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
```

Unity environments contain **brains**, our interfaces for controlling agents. We'll be conrtolling the first (default) brain in the environment. It's also useful to keep information such as `state_size`, `action_size` and `num_agents`.

```python
env = None
system = platform.system()
if system == 'Linux':
    env = UnityEnvironment(file_name="setup/Reacher_Linux/Reacher.x86_64")
elif system == 'Darwin':
    env = UnityEnvironment(file_name="setup/Reacher.app")
elif system == 'Windows':
    env = UnityEnvironment(file_name="setup/Reacher_Windows_x86_64/Reacher.exe")
else:
    print('Cannot find environment for this system.')

# use the default brain
brain_name = env.brain_names[0]
brain = env.brains[brain_name]

# reset the environment
env_info = env.reset(train_mode=True)[brain_name]
num_agents = len(env_info.agents)
action_size = brain.vector_action_space_size
state_size = env_info.vector_observations.shape[1]
```

    INFO:unityagents:
    'Academy' started successfully!
    Unity Academy name: Academy
            Number of Brains: 1
            Number of External Brains : 1
            Lesson number : 0
            Reset Parameters :
    		goal_size -> 5.0
    		goal_speed -> 1.0
    Unity brain name: ReacherBrain
            Number of Visual Observations (per agent): 0
            Vector Observation space type: continuous
            Vector Observation space size (per agent): 33
            Number of stacked Vector Observation: 1
            Vector Action space type: continuous
            Vector Action space size (per agent): 4
            Vector Action descriptions: , , ,

## 2. Actor

A2C is an **actor-critic method** and as such requires two models, one to represent the policy (the actor) and one to represent the value function (the critic). These models could be represented with just one network that has two distinct final layers – this is particularly useful when there are convolutional layers at the beginning of the network, as it helps extrapolating important information from the environment – but we don't need to do that, since the input consists of only 33 values per agent. We'll instead implement two separate models.

The `Actor` class is straightforward and consists of only fully connected layers:

- `state_size` $\to 64$ followed by `relu` activations
- $64 \to 128$ and `relu`
- $128 \to 64$ and `relu`
- $64 \to \mu$, that is $64 \to$ `action_size`.

Another important aspect of the actor is that it should output a normal distribution, so $\mu$ is not enough. To output $\sigma$ I decided to create a dedicated `nn.Parameter` of size `action_size` that does not depende on the input, although can still be updated by autograd.

In this implementation, **the actor is always used in a batch**. There are 20 independent agents that share the same network, so the input is made of 20 vectors for a total size of `[num_agents, state_size]` and, similarly, the output contains 20 $\mu$ vectors that can be seen as a matrix of size `[num_agents, action_size]`. Note that $\sigma$ does not change with the number of agents, since it will be applied equally to all `torch.distributions` objects.

```python
class Actor(nn.Module):

    def __init__(self, state_size, action_size, hidden_layers=[64, 128, 64]):
        super(Actor, self).__init__()
        self.action_size = action_size

        # prepare the first hidden layer
        self.hidden_layers = nn.ModuleList([nn.Linear(state_size, hidden_layers[0])])

        # prepare the rest of the hidden layers
        A = hidden_layers[:-1]
        B = hidden_layers[1:]
        self.hidden_layers.extend([nn.Linear(a, b) for a, b in zip(A, B)])

        # the actor will output the parameters of a normal distribution,
        self.mu_layer = nn.Linear(hidden_layers[-1], action_size)
        self.sigma = nn.Parameter(torch.ones(1, action_size))

    def forward(self, state):
        for layer in self.hidden_layers:
            state = layer(state)
            state = F.relu(state)
        mu = self.mu_layer(state)
        return mu, self.sigma
```

## 3. Critic

The `Critic` class is also made of only fully connected layers:

- `state_size` $\to 64$ followed by `relu` activations
- $64 \to 128$ and `relu`
- $128 \to 64$ and `relu`
- $64 \to V(s)$, that is $64 \to 1$ and no activation.

`Critic` is very similar to the `DQN` class, except it does not process the value for state-action pairs $Q(s, a)$, but only the value of the state $V(s)$. The reason for this is that we don't care about the value of an individual state-action pair because we're going to use the critic to calculate **advantages**, and based on that we'll tweak the probability distribution that the actor generates.

As for the actor, **the critic is always used in batches** of 20 independent states, for a total size of `[num_agents, state_size]` and. Similarly, the output contains 20 $V(s)$ values that can be seen as a matrix of size `[num_agents, 1]`.

```python
class Critic(nn.Module):

    def __init__(self, state_size, hidden_layers=[64, 128, 64]):
        super(Critic, self).__init__()

        # prepare the first hidden layer
        self.hidden_layers = nn.ModuleList([nn.Linear(state_size, hidden_layers[0])])

        # prepare the rest of the hidden layers
        A = hidden_layers[:-1]
        B = hidden_layers[1:]
        self.hidden_layers.extend([nn.Linear(a, b) for a, b in zip(A, B)])

        # the critic outputs only a scalar V(s)
        self.value_layer = nn.Linear(hidden_layers[-1], 1)

    def forward(self, state):
        # connect layers to each other and put relu activations between them
        for layer in self.hidden_layers:
            state = layer(state)
            state = F.relu(state)
        value = self.value_layer(state)
        return value
```

## 4. Advantage estimation

As mentioned, the critic is used to calculate advantages. An advantage value gives us a way to evaluate an action. For example, given a tuple $(s, a, r, s')$, as an indication for the goodness of the action $a$, we can calculate the advantage $A = r + V(s') - V(s)$, which is hopefully positive, but could also be negative. Since the output of the actor is a probability distribution, we can use **gradient ascent to increase** the log probability $\nabla_\theta \ln \pi_\theta(a | s)$ of that action being taken, and if we change the sign we will **decrease** it instead.

Combining the two ideas in a single expression, we get $\nabla_\theta \ln \pi_\theta(a | s) \cdot A$. This is how we'll decide whether we'll increase or decrease the probability of the action $a$, and by how much. Of course, in order to do that, our estimate of $V$, and consequently of $A$ must be accurate. The following sections describe two ways to obtain these values, based on the fact that A2C works using $n$-step TD estimates.

### 4.1 $n$-step

The overline brackets highlights how $V$ is updated and how the advantage is estimated. In fact, when updating $V(s_{i+n-1})$ we consider the standard one-step TD error that only includes $R_{i+n}$ and $V(s_{i+n})$. However, in order to improve our next estimates, we now substitute $V(s_{i+n-1})$ with $R_{i+n} + \gamma V(s_{i+n})$, so that the update to $V(s_{i+n-2})$ will be a two-step TD error. I decided to include $G(s_{i+n+1})$ in the image to highlight that there is only one bootstrapping step in this process. The same reasoning is used to estimate the advantages. For example, if $n=3$ we have
$$
\begin{align}
\begin{split}
V(s_{i+2}) \leftarrow & R_{i+3} + \gamma V(s_{i+3}) \\
V(s_{i+1}) \leftarrow & R_{i+2} + \gamma R_{i+3} + \gamma^2 V(s_{i+3}) \\
V(s_{i}) \leftarrow & R_{i+1} + \gamma R_{i+2} + \gamma^2 R_{i+3} + \gamma^3 V(s_{i+3})
\end{split}
\qquad\qquad
\begin{split}
A(s_{i+3}) = & R_{i+3} + \gamma V(s_{i+3}) - V(s_{i+2}) \\
A(s_{i+2}) = & R_{i+2} + \gamma R_{i+3} + \gamma^2 V(s_{i+3}) - V(s_{i+1}) \\
A(s_{i+1}) = & R_{i+1} + \gamma R_{i+2} + \gamma^2 R_{i+3} + \gamma^3 V(s_{i+3}) - V(s_{i})
\end{split}
\end{align}
$$
When we take into account that the critic is updated according to the mean squared error, both equations become virtually the same thing, as they are both implemented using discounted returns.

The next two functions, `n_step` and `gae`, take more parameters than they need. Technically, both of them need `rewards, dones, future, critic_out, gamma`, with `gae` also requiring another hyperparameter `tau` that will be explained in the next markdown cell. However, `n_step` and `gae` only affect how advantages are calculated, but we're always going to compute `returns` regardless of this choice, since we need it for the critic. For this reason, I decided to also include `returns` as a parameter, so that we only need to calculate it once. The implementation of `n_step` then becomes trivial.

```python
def n_step(rewards, dones, future, returns, critic_out, gamma, tau):
    return returns - critic_out
```

### 4.2 GAE

As already mentioned, the way $V$ is updated is identical in both `n_step` and `gae`, so I left the overline brackets in the image to highlight this fact. On the other hand, the green and orange blocks indicate how the $\hat{A}$ values are computed. These values are not the actual advantages $A$ that `gae` estimates, but they help us get there – except for the last block, where $A$ is the same as $\hat{A}$. For instance, if $n = 3$ we have
$$
\begin{align}
\begin{split}
\hat{A}(s_{i+3}) &= R_{i+3} + \gamma V(s_{i+3}) - V(s_{i+2})\\
\hat{A}(s_{i+2}) &= R_{i+2} + \gamma V(s_{i+2}) - V(s_{i+1})\\
\hat{A}(s_{i+1}) &= R_{i+1} + \gamma V(s_{i+1}) - V(s_{i})
\end{split}
\qquad\qquad
\begin{split}
A(s_{i+3}) &= \hat{A}(s_{i+3})\\
A(s_{i+2}) &= \hat{A}(s_{i+2}) + \gamma \lambda A(s_{i+3})\\
A(s_{i+1}) &= \hat{A}(s_{i+1}) + \gamma \lambda A(s_{i+2})\\
\end{split}
\end{align}\\
$$
Essentially, $\hat{A}$ represents a one-step advantage (as indicated by the contents of the green and orange blocks in the image), while $A$ is a mixture of steps (or blocks in the image), controlled by $\lambda$. Indeed, if we expand $A(s_{i+2})$ we can see that it contains a little bit of both blocks
$$
A(s_{i+2}) = R_{i+2} + \gamma V(s_{i+2}) - V(s_{i+1}) + \gamma \lambda (R_{i+3} + \gamma V(s_{i+3}) - V(s_{i+2}))\\= R_{i+2} + \gamma\lambda R_{i+3} + (1 - \lambda)\gamma V(s_{i+2}) + \gamma^2\lambda V(s_{i+3}) - V(s_{i+1})
$$
Additionally, if $\lambda = 0$ the expression simplifies to a one-step TD approach, whereas with $\lambda = 1$ it becomes like a MC approach
$$
\begin{align}
\lambda = 0 \implies \quad & R_{i+2} + \gamma V(s_{i+2}) - V(s_{i+1})\\
\lambda = 1 \implies \quad & R_{i+2} + \gamma R_{i+3} + \gamma^2 V(s_{i+3}) - V(s_{i+1})
\end{align}
$$
More generally, this is called the $\lambda$-return and is often indicated with TD($\lambda$) – although we'll refer to its hyperparameter as `tau`, because `lambda` is a reserved keyword in python.

```python
def gae(rewards, dones, future, returns, critic_out, gamma, tau):

    # create a single tensor for rewards, dones and predicted_values.
    # also, dones and predicted_values have n+1 rows: predictied_values adds G or future at the end
    # whereas dones adds a row of zeros at the beginning; this simplifies access to this data in the loop
    rewards = torch.cat([r.unsqueeze(0) for r in rewards], dim=0)
    dones = torch.cat([torch.zeros([1, 20, 1]).to(device)] + [d.unsqueeze(0) for d in dones], dim=0)
    predictied_values = torch.cat((critic_out, future.unsqueeze(0)), dim=0)

    # retrieve n from a tensor's size, instead of passing it as a parameter
    n = list(critic_out.size())[0]

    # compute A_hats and As backwards, and store the As as you go
    A = torch.zeros([20, 1]).to(device)
    advantages = []
    for i in reversed(range(n)):
        A_hat = rewards[i] + gamma * predictied_values[i+1] * (1 - dones[i+1]) - predictied_values[i] * (1 - dones[i])
        A = A_hat + gamma * tau * A
        advantages.insert(0, A.unsqueeze(0))
    return torch.cat(advantages, dim=0)
```

## 5. Agent

As stated, the learning algorithm guides the actor with gradient ascent on $\nabla_\theta \ln \pi_\theta(a | s) \cdot A$ and the critic with gradient descent on the mean squared error of returns. To implement this, we can use a **temporary buffer** somewhat similar to the one found in DQN or DDPG, except it is reset after every learning step and there is no sampling involved, so the size will always be limited by `n`. Thus, we can avoid building a dedicated class for it and instead make a few python arrays. The ones that we need are `tmp_r` and `tmp_d`, respectively for storing rewards and the done flag. We do not need to store the actual states and actions, as long as we store their values and log probabilities, respectively. To do this, we use `tmp_critic_out` and `tmp_log_prob`. Finally, we'll need `tmp_ns` to store the `n`-th next state, that we'll use for bootstrapping.

There are very few hyperparameters compared to other methods. What we need is:
- `n` determines how many steps to take for one learning step; `1` is similar to other standard TD methods, while higher values make it more like a MC method
- `calc_advantages` can be either `n_step` or `gae`
- `alpha` is the learning rate of both networks
- `gamma` is the discount factor for the returns
- `tau` in `[0, 1]` will be used only if `calc_advantages` is `gae` and determines how to mix the advantages; `0` makes it a standard TD method, while `1` makes it similar to a MC method, except it is still limited by `n`

The agent has three main functions, but only two of them should be called from the outside:
- `act`, based on a state tensor `state`, choose an action. This also evaluates the state and the log probability of the action, as well as stores some information.
- `store`, receive a complete tuple and adds some data to the temporary buffer. This counts as a step towards the updates of `actor` and `critic`, and in fact calls `learn()` automatically when needed.
- `learn`, takes all the data in the buffer and calculates discounted returns, advantages using `calc_advantages`, and performs the update of the networks.

Additionally, an agent can be `reset()`.

```python
class A2CAgent:

    # -- initialization -- #

    def __init__(self, state_size, action_size, calc_advantages, n=4,
                 alpha=0.0001, gamma=0.95, tau=0.5):

        self.state_size, self.action_size = state_size, action_size
        self.alpha, self.gamma, self.tau = alpha, gamma, tau
        self.calc_advantages = calc_advantages
        self.n = n
        self.reset()

    def reset_temporary_buffer(self):
        # some values will be stored in temporary buffers to
        # avoid re-evaluating them and to speed up the execution
        self.tmp_r, self.tmp_ns, self.tmp_d = [], None, []
        self.tmp_log_prob, self.tmp_critic_out = [], []

    def reset(self):
        self.actor = Actor(self.state_size, self.action_size).to(device)
        self.actor_optimizer = optim.Adam(self.actor.parameters(), lr=self.alpha)

        self.critic = Critic(self.state_size).to(device)
        self.critic_optimizer = optim.Adam(self.critic.parameters(), lr=self.alpha)

        self.reset_temporary_buffer()
        self.i = 0

    # -- initialization -- #

    def act(self, state):
        state = torch.FloatTensor(state).to(device)

        # evaluate the current state and choose
        #  - its value
        value = self.critic(state)

        #  - an action from its probability distribution
        mu, sigma = self.actor(state)
        dist = torch.distributions.Normal(mu, sigma)
        action = dist.sample().detach()

        # then also evaluate the log probability of choosing that action.
        # this is what we'll use for backpropagation, together with the estimated advantage
        log_prob = dist.log_prob(action)

        # store these values in a temporary buffer so that we don't need to calculate them again.
        # this is like a replay buffer in DQN, but it gets wiped after an update step.
        self.tmp_log_prob.append(log_prob.unsqueeze(0))
        self.tmp_critic_out.append(value.unsqueeze(0))

        # note: clipping the action to [-1, 1] is required by the environment, but in my tests,
        # clipping it before calculating its log probability resulted in poor performance
        return np.clip(np.array(action), -1, 1)

    def store(self, s, a, r, ns, d):
        # ignore s and a because they're already handled in act(state)
        self.tmp_r.append(torch.FloatTensor(r).unsqueeze(1).to(device))
        self.tmp_ns = torch.FloatTensor(ns).to(device)
        self.tmp_d.append(torch.FloatTensor(d).unsqueeze(1).to(device))

        # every n steps we should perform an update step
        if self.i == self.n - 1:
            self.learn()
        self.i = (self.i + 1) % self.n

    def learn(self):
        # note that this is called automatically by the agent

        # create a single tensor for the values stored in act(), namely tmp_log_prob and tmp_critic_out
        log_prob = torch.cat(self.tmp_log_prob, dim=0)
        critic_out = torch.cat(self.tmp_critic_out, dim=0)

        # calculate discounted rewards backwards, starting from G or future (the bootstrapping step)
        future = self.critic(self.tmp_ns)
        tmp_returns = []
        ret = future
        for reward, done in zip(reversed(self.tmp_r), reversed(self.tmp_d)):
            ret = reward + self.gamma * (1 - done) * ret
            tmp_returns.insert(0, ret.unsqueeze(0))
        returns = torch.cat(tmp_returns, dim=0).to(device)

        # use either n_step or gae to estimate the advantages.
        # note that not all of these parameters are guaranteed to be used by the given function.
        advantages = self.calc_advantages(self.tmp_r, self.tmp_d, future, returns, critic_out, self.gamma, self.tau)

        # train the actor by guiding the increase or decrease of log_prob with the advantages.
        # note the negative sign: pytorch executes gradient descent, so we want to invert that to gradient ascent.
        policy_loss = -torch.mean(log_prob * advantages.detach())
        self.actor_optimizer.zero_grad()
        policy_loss.backward()
        nn.utils.clip_grad_norm_(self.actor.parameters(), 5)  # the use of gradient clipping helped in my tests
        self.actor_optimizer.step()

        # train the critic on the mean squared error between its output and the discounted returns
        value_loss = torch.mean((returns.detach() - critic_out) ** 2)
        self.critic_optimizer.zero_grad()
        value_loss.backward()
        nn.utils.clip_grad_norm_(self.critic.parameters(), 5)  # the use of gradient clipping helped in my tests
        self.critic_optimizer.step()

        # finally empty the temporary buffer
        self.reset_temporary_buffer()
```

## 6. Training an agent

Creating an agent is straightforward now. We can make three agents, with some combination of `n_step`, `gae`, and `n`, to see how these parameters affect learning.

- `alpha=0.0001`
- `gamma=0.95`
- `tau=0.5`

```python
agents = [('A2Cn4', A2CAgent(state_size, action_size, n_step, n=4)),
          ('A2Cn8', A2CAgent(state_size, action_size, n_step, n=8)),
          ('A2C-GAEn8', A2CAgent(state_size, action_size, gae, n=8))]
```

What we define now is a function that connects our `agent` and `env`. Recall that we'll be using the default `brain_name` to interact with the environment. The agent is allowed to store observations and, consequently, learn every `n` steps.

```python
def execute_episode(agent, env):
    # prepare the environment
    scores = np.zeros(num_agents)
    env_info = env.reset(train_mode=True)[brain_name]

    # get the initial state
    states = env_info.vector_observations
    while True:

        # evaluate the current state
        actions = agent.act(states)

        # execute the chosen action and get the outcome
        env_info = env.step(actions)[brain_name]
        next_states = env_info.vector_observations
        rewards = env_info.rewards
        dones = env_info.local_done

        # store the experience (also automatically learn, from time to time)
        agent.store(states, actions, rewards, next_states, dones)

        # prepare for the next iteration
        states = next_states
        scores += rewards

        if np.any(dones):
            break

    # return the total rewards obtained
    return np.mean(scores)
```

We can call the function above repeatedly while **monitoring** what happens, so that we'll be able to compare different approaches. The most important indicator is going to be the average score over 100 consecutive episodes (where each episode's score is the average of its 20 independent agents): the environment is considered **solved when the agents scores +30 or more** under this metric. We'll be training each variation 3 times from scratch, for 600 episodes, and we'll average the results. This way, the learning curves will depend less on the randomness of the environment and more on the actual differences of the proposed approaches.

```python
def train(agent, env, episodes=600, repeat=3, consecutive_episodes=100, show_output=True, save_as=None):

    results = [None] * repeat
    for r in range(repeat):
        partial_results = [None] * episodes

        # reset the agent to start learning from scratch
        agent.reset()
        for i in range(episodes):

            # execute all the episodes and store the results
            score = execute_episode(agent, env)
            partial_results[i] = score

            if show_output:
                print("\r[{}] Episode: {}, Score: {}".format(r+1, i+1, score), end="")
                sys.stdout.flush()
        if show_output:
            print()

        results[r] = partial_results

        # store the trained model if requested
        if save_as is not None:
            torch.save(agent.actor.state_dict(), '{}/{}_actor.pth'.format(save_as, r+1))
            torch.save(agent.critic.state_dict(), '{}/{}_critic.pth'.format(save_as, r+1))
            np.save('{}/report_{}'.format(save_as, r+1), np.array(partial_results))

    results = np.array(results)
    if save_as is not None:
        np.save('{}/full_report'.format(save_as), np.array(results))

    # use convolutions to calculate the mean and standard deviation summarizing the training step
    mean = signal.convolve2d(results, np.ones([repeat, consecutive_episodes]) / (repeat * consecutive_episodes), mode='valid')
    v = signal.convolve2d(results, np.ones([1, consecutive_episodes]) / consecutive_episodes, mode='valid')
    std_dev = signal.convolve2d(v ** 2 - mean ** 2, np.ones([repeat, 1]) / repeat, mode='valid') ** (1/2)
    return mean.flatten(), std_dev.flatten(), results
```

```python
training_results = []
for ag in agents:
    name, agent = ag
    print('Agent: {}'.format(name))
    mean, std_dev, full_report = train(agent, env, save_as='models/{}'.format(name.replace(" ", "_")))
    training_results.append((mean, std_dev, full_report))
env.close()
```

    Agent: A2Cn4
    [1] Episode: 600, Score: 39.00949912806973
    [2] Episode: 600, Score: 39.3239991210401
    [3] Episode: 600, Score: 39.14199912510812
    Agent: A2Cn8
    [1] Episode: 600, Score: 39.081999126449226
    [2] Episode: 600, Score: 37.057999171689154
    [3] Episode: 600, Score: 39.247499122750014
    Agent: A2C-GAEn8
    [1] Episode: 600, Score: 38.7974991328083
    [2] Episode: 600, Score: 39.176499124336985
    [3] Episode: 600, Score: 38.591499137412754

## 7. Comparing the results

We can use `matplotlib` to plot the mean of the score. All proposed variants manage to reach the goal of +30, in fact they almost get +40 points. However, there are some notable differences between them.

First, we can see that, for this task, the A2C method with `n = 8` is the first one to reach the goal, solving the environment in 314 episodes, followed by the `n = 4` version and then finally by the GAE version with `n = 8`. However, we can't yet say that the best approach is A2C with `n = 8` and in fact, if we look at its stability (the orange shadow), it clearly exhibits a higher variance after 600 episodes compared to the other two variants.

This seems in line with the fact that Monte Carlo approaches usually have a higher variance, and A2C with `n = 8` is closer to MC than A2C with `n = 4`. As for the remaining variant with GAE, although it also has `n = 8`, it does not show such high variance at the end of the learning process. This may be its strenght: instead of being the fastest method, it may allow for a more flexible and easy choice of `n` (which is notably hard in some tasks) while retaining good convergence properties.

```python
def plot_results(agents, training_results):
    for agent, ts in zip(agents, training_results):
        mean, std_dev, _ = ts
        episode_solved = np.argmax(mean > 30) + 100
        print('{}: Solved after {} episodes'.format(agent[0], episode_solved))

        x = np.arange(100, 100 + mean.size)
        line = plt.plot(x, mean, label = agent[0])
        plt.fill_between(x, mean + std_dev, mean - std_dev, alpha=0.15)
        plt.axvline(episode_solved, lw=0.75, color=line[0].get_color())

    plt.axhline(30, dashes=(4, 8), lw=0.75, color='black')
    plt.xticks(np.arange(1, 1 + mean.size / 100) * 100)
    plt.yticks(np.arange(0, 9) * 5)
    plt.xlabel('Number of episodes')
    plt.title('Average score over 100 consecutive episodes')
    plt.legend()
    plt.show()

plot_results(agents, training_results)
```

## 8. Possible improvements

One of the things that is usually applied to policy gradient methods is an **entropy** term to help with exploration. In fact, given that the actor outputs probabilities, it is straightforward to add an entropy term to its loss so that it will be maximized. In particular, since we're using a normal distribution here, the ouptut of the model only consists of its parameters, $\mu$ and $\sigma$, and maximizing the entropy is essentially the same as maximizing $\sigma$, or in other words, the randomness of the agent. The entropy term should always be limited by an hyperparameters, otherwise the actor won't be able to converge to any policy.

It would be interesting to compare this solution to other policy gradient algorithms like [REINFORCE](https://link.springer.com/content/pdf/10.1007/BF00992696.pdf), [PPO](https://arxiv.org/pdf/1707.06347.pdf) or [DDPG](https://arxiv.org/pdf/1509.02971.pdf), or even to black-box optimization techniques like [ES](https://arxiv.org/pdf/1703.03864.pdf).

## 9. Conclusions

For this task, the agents proposed manage to play well every time. The models used to create the graphs in section 7 are available in the repository. The full documented code is also available and `main.py` can be executed to train or load a pre-trained A2C-GAE agent with `n = 8` and see it in action, just like it was done to get the model `final.pht`.


```python
clean_up=True # removes gams-related files in work-folder if true
%run StdPackages.ipynb
os.chdir(directory['py'])
import techdata_to_tree, sys, ShockFunction
import abatement_EOP as abatement
os.chdir(directory['curr'])
def flatten_list(list_):
    return [item for sublist in list_ for item in sublist]
```

*Specify settings:*

```python
name = 'A1_test'
f = {'tech': directory['data']+"/techdata_dors_2.xlsx",
     'other': directory['data']+"/Othdata_dors_2.xlsx"}
s = ['inputdisp','endofpipe','inputprices']
m = ['ID','EOP']
e = {'M': 'vars','M_sets': 'sets'} # dictionary used to load emissions data
other_trees = {'ID_Y': {'s': 'Y', 'read_data': {'Y': 'vars'}, 'Q2P': 'Q2P'}} # the key = tree name, s = sheet that specifies tree structure, read_data (if not None) specifies how to load data.
```

```python
type_f = {'ID_EC': 'CES_norm',
          'ID_CU': 'MNL',
          'ID_TU': 'CET_norm',
          'ID_TX': 'CES',
          'ID_BU': 'linear_out',
          'ID_BX': 'CES',
          'EOP_CU': 'MNL',
          'EOP_TU': 'CET_norm',
          'EOP_TX': 'CES'}
```

*Set up nesting tree:*

```python
%run ATree.ipynb
```

## 1: Initialize model

*Set up gams model from tree:*

```python
gm = abatement.abate(nts=nts,tech=tech,work_folder=directory['work'],**{'data_folder':directory['gams'],'name': name})
```

*Add a couple of extra aliases, and update sets with elements:*

```python
gm.add_aliases([('n','nnnn'),('n','nnnnn')])
gm.model.database.update_all_sets(clean_up=False)
```

*Initialise variables at leontief levels, and fill out with simple standard values in potentially missing domains:*

```python
gm.initialize_variables_leontief()
gm.initialize_variables(**{'check_variables': True})
```

## 2: Baseline solution

*Set up model and solve:*

```python
gm.write_and_run(name='baseline')
db0 = gm.model_instances['baseline'].out_db
```

*Print some?*

```python
var = 'PwThat'
db0.get(var).plot.bar();
```

## A: Some debugging functions if there are problems with groups

*Debugging of initial states:*

```python
debug = False
if debug is True:
    gm.initialize_variables(**{'check_variables': True})
    from IPython.display import Markdown
    # Test groupings:*
    var = gm.default_variables[1]
    display(Markdown(f"**{var}+' exogenous:'**"))
    display(gm.var_exo(var))
    display(Markdown(f"**{var}+' endogenous:'**"))
    display(gm.var_endo(var))
```

*Debugging of writing groups:*

```python
group = gm.model.settings.name+'_'+'g_ID_alwaysendo'
debug_group = False
if debug_group is True:
    gm.initialize_variables(**{'check_variables':True})
    if group in gm.exo_groups:
        group_vals = gm.exo_groups[group] # corresponding group conditions
    else:
        group_vals = gm.endo_groups[group]
    gm.model.groups[group] = group_vals # add to model.groups
    gm.model.settings.get_conf('g_exo').update([group]) # add to exogenous groups
    repo = None
    self = gm.model
    if repo is None:
        repo = self.settings.data_folder
    if not os.path.exists(repo):
        os.makedirs(repo)
    self.write_default_components()
    self.default_export(repo)
```

## B: Some details on added modules

### ID sums:

* Define the output share os[n,nn] as the share of output that technology $nn$ produces of energy-service type $n$. This should be defined as:

$$
\begin{align}
    \text{os[n,nn]} = \text{sum(nnn\$(e2u[n,nnn] and u2t[nnn,nn]), qD[nnn]) / qD[nn]}.
\end{align}
$$

* Define variable qSumX[n,nn] defined as the use of aggregate input type $nn\in (electricity,oil,inp3,K)$ used to produce energy service $n$. This is defined as:

$$\begin{align}
    \text{qSumX[n,nn]} = \text{sum([nnn,nnnn]\$(e2ai2i[n,nn,nnn] and e2t[n,nnnn] and i2t[nnn,nnnn]), qD[nnn] * os[n,nnnn])}
\end{align}$$
* Here e2ai2i identifies the relevant quantities $qD[i]$ that are tied to relevant aggregate type of input, $e2t[e,t]$ identifies the correct output-share $os[t,e]$, and $i2t[i,t]$ ensures that only the relevant $i,t$ combinations are summed over.

### ID emissions:

Emissions accounting with only the input displacing module:
* $M0[z]$ defines pre-emission abatement (equivalent to emissions without EOP module). This is defined over z (emission types).
* $phi[z,n]$ defines mission intensities over aggregate input types ($n\in ai[n]$).
Pre abatement emissions are then defined as:

$$\begin{align}
    M0[z] = sum(n$(ai[n]) phi[z,n] * qD[n]),
\end{align}$$
* As long as the equation for the aggregate use of inputs in the sector (qD[n] for ai[n]) takes into account which modules are active, this equation stays the same across all states.

* $PwThat[n]\$(ID\_inp[n]):$ defines the effective prices on inputs, taking the price on emissions taxes and abatement into account:

$$\begin{align}
    PwThat[n] = PwT[n] + sum(z, phi[z,n] * pMhat[z]);
\end{align}$$

* As long as we establish an equation for pMhat[z] (corrected emission prices) that depends on the type of state the model is in, this equation stays the same across all states. However, with the EOP module activated, there should be an equivalent equation defined for ($EOP\_inp[n]$ and not $ID\_inp[n]$).

### ID Aggregates:

* If the state is ID the aggregate use of inputs ($qD[n]\$(ai[n])$) are defined from (we could let the namespace for 'ai' look up 'ID\_ai' instead of 'ai' when the EOP set is not activated?)

$$\begin{align}
    qD[n] = sum(nn\$(ID\_i2ai[nn,n]), qD[nn])
\end{align}$$

* Without the EOP module, a second equation simply states that $pMhat = pM$.


## Configurations for Colab

```python
import sys
IN_COLAB = 'google.colab' in sys.modules

if IN_COLAB:
    !apt-get install -y xvfb python-opengl > /dev/null 2>&1
    !pip install gym pyvirtualdisplay > /dev/null 2>&1
    !pip install JSAnimation==0.1
    !pip install pyglet==1.3.2

    from pyvirtualdisplay import Display

    # Start virtual display
    dis = Display(visible=0, size=(400, 400))
    dis.start()
```

# 05. Noisy Networks for Exploration

[M. Fortunato et al., "Noisy Networks for Exploration." arXiv preprint arXiv:1706.10295, 2017.](https://arxiv.org/pdf/1706.10295.pdf)

NoisyNet is an exploration method that learns perturbations of the network weights to drive exploration. The key insight is that a single change to the weight vector can induce a consistent, and potentially very complex, state-dependent change in policy over multiple time steps.

Firstly, let's take a look into a linear layer of a neural network with $p$ inputs and $q$ outputs, represented by

$$
y = wx + b,
$$

where $x \in \mathbb{R}^p$ is the layer input, $w \in \mathbb{R}^{q \times p}$, and $b \in \mathbb{R}$ the bias.

The corresponding noisy linear layer is defined as:

$$
y = (\mu^w + \sigma^w \odot \epsilon^w) x + \mu^b + \sigma^b \odot \epsilon^b,
$$

where $\mu^w + \sigma^w \odot \epsilon^w$ and $\mu^b + \sigma^b \odot \epsilon^b$ replace $w$ and $b$ in the first linear layer equation. The parameters $\mu^w \in \mathbb{R}^{q \times p}, \mu^b \in \mathbb{R}^q, \sigma^w \in \mathbb{R}^{q \times p}$ and $\sigma^b \in \mathbb{R}^q$ are learnable, whereas $\epsilon^w \in \mathbb{R}^{q \times p}$ and $\epsilon^b \in \mathbb{R}^q$ are noise random variables which can be generated by one of the following two ways:

1. **Independent Gaussian noise**: the noise applied to each weight and bias is independent, where each random noise entry is drawn from a unit Gaussian distribution. This means that for each noisy linear layer, there are $pq + q$ noise variables (for $p$ inputs to the layer and $q$ outputs).
2. **Factorised Gaussian noise:** This is a more computationally efficient way. It produces 2 random Gaussian noise vectors ($p, q$) and makes $pq + q$ noise entries by outer product as follows:

$$
\begin{align}
\epsilon_{i,j}^w &= f(\epsilon_i) f(\epsilon_j),\\
\epsilon_{j}^b &= f(\epsilon_i),\\
\text{where } f(x) &= sgn(x) \sqrt{|x|}.
\end{align}
$$

In all experiements of the paper, the authors used Factorised Gaussian noise, so we will go for it as well.

```python
import math
import os
from typing import Dict, List, Tuple

import gym
import matplotlib.pyplot as plt
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from IPython.display import clear_output
```

## Replay buffer

Please see *01.dqn.ipynb* for detailed description.

```python
class ReplayBuffer:
    """A simple numpy replay buffer."""

    def __init__(self, obs_dim: int, size: int, batch_size: int = 32):
        self.obs_buf = np.zeros([size, obs_dim], dtype=np.float32)
        self.next_obs_buf = np.zeros([size, obs_dim], dtype=np.float32)
        self.acts_buf = np.zeros([size], dtype=np.float32)
        self.rews_buf = np.zeros([size], dtype=np.float32)
        self.done_buf = np.zeros(size, dtype=np.float32)
        self.max_size, self.batch_size = size, batch_size
        self.ptr, self.size, = 0, 0

    def store(
        self,
        obs: np.ndarray,
        act: np.ndarray,
        rew: float,
        next_obs: np.ndarray,
        done: bool,
    ):
        self.obs_buf[self.ptr] = obs
        self.next_obs_buf[self.ptr] = next_obs
        self.acts_buf[self.ptr] = act
        self.rews_buf[self.ptr] = rew
        self.done_buf[self.ptr] = done
        self.ptr = (self.ptr + 1) % self.max_size
        self.size = min(self.size + 1, self.max_size)

    def sample_batch(self) -> Dict[str, np.ndarray]:
        idxs = np.random.choice(self.size, size=self.batch_size, replace=False)
        return dict(obs=self.obs_buf[idxs],
                    next_obs=self.next_obs_buf[idxs],
                    acts=self.acts_buf[idxs],
                    rews=self.rews_buf[idxs],
                    done=self.done_buf[idxs])

    def __len__(self) -> int:
        return self.size
```

## Noisy Layer

**References:**
- https://github.com/higgsfield/RL-Adventure/blob/master/5.noisy%20dqn.ipynb
- https://github.com/Kaixhin/Rainbow/blob/master/model.py

```python
class NoisyLinear(nn.Module):
    """Noisy linear module for NoisyNet.

    Attributes:
        in_features (int): input size of linear module
        out_features (int): output size of linear module
        std_init (float): initial std value
        weight_mu (nn.Parameter): mean value weight parameter
        weight_sigma (nn.Parameter): std value weight parameter
        bias_mu (nn.Parameter): mean value bias parameter
        bias_sigma (nn.Parameter): std value bias parameter

    """

    def __init__(self, in_features: int, out_features: int, std_init: float = 0.5):
        """Initialization."""
        super(NoisyLinear, self).__init__()

        self.in_features = in_features
        self.out_features = out_features
        self.std_init = std_init

        self.weight_mu = nn.Parameter(torch.Tensor(out_features, in_features))
        self.weight_sigma = nn.Parameter(
            torch.Tensor(out_features, in_features)
        )
        self.register_buffer(
            "weight_epsilon", torch.Tensor(out_features, in_features)
        )

        self.bias_mu = nn.Parameter(torch.Tensor(out_features))
        self.bias_sigma = nn.Parameter(torch.Tensor(out_features))
        self.register_buffer("bias_epsilon", torch.Tensor(out_features))

        self.reset_parameters()
        self.reset_noise()

    def reset_parameters(self):
        """Reset trainable network parameters (factorized gaussian noise)."""
        mu_range = 1 / math.sqrt(self.in_features)
        self.weight_mu.data.uniform_(-mu_range, mu_range)
        self.weight_sigma.data.fill_(
            self.std_init / math.sqrt(self.in_features)
        )
        self.bias_mu.data.uniform_(-mu_range, mu_range)
        self.bias_sigma.data.fill_(
            self.std_init / math.sqrt(self.out_features)
        )

    def reset_noise(self):
        """Make new noise."""
        epsilon_in = self.scale_noise(self.in_features)
        epsilon_out = self.scale_noise(self.out_features)

        # outer product
        self.weight_epsilon.copy_(epsilon_out.ger(epsilon_in))
        self.bias_epsilon.copy_(epsilon_out)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """Forward method implementation.

        We don't use separate statements on train / eval mode.
        It doesn't show remarkable difference of performance.
        """
        return F.linear(
            x,
            self.weight_mu + self.weight_sigma * self.weight_epsilon,
            self.bias_mu + self.bias_sigma * self.bias_epsilon,
        )

    @staticmethod
    def scale_noise(size: int) -> torch.Tensor:
        """Set scale to make noise (factorized gaussian noise)."""
        x = torch.FloatTensor(np.random.normal(loc=0.0, scale=1.0, size=size))

        return x.sign().mul(x.abs().sqrt())
```

## Noisy Network

We use NoisyLinear for the last two FC layers, and there is a method to reset noise at every step.
These are the only differences from the example of *01.dqn.ipynb*.

```python
class Network(nn.Module):
    def __init__(self, in_dim: int, out_dim: int):
        """Initialization."""
        super(Network, self).__init__()

        self.feature = nn.Linear(in_dim, 128)
        self.noisy_layer1 = NoisyLinear(128, 128)
        self.noisy_layer2 = NoisyLinear(128, out_dim)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """Forward method implementation."""
        feature = F.relu(self.feature(x))
        hidden = F.relu(self.noisy_layer1(feature))
        out = self.noisy_layer2(hidden)

        return out

    def reset_noise(self):
        """Reset all noisy layers."""
        self.noisy_layer1.reset_noise()
        self.noisy_layer2.reset_noise()
```

## DQN + NoisyNet Agent (w/o DuelingNet)

Here is a summary of DQNAgent class.

| Method           | Note                                                 |
| ---              | ---                                                  |
|select_action     | select an action from the input state.               |
|step              | take an action and return the response of the env.   |
|compute_dqn_loss  | return dqn loss.                                     |
|update_model      | update the model by gradient descent.                |
|target_hard_update| hard update from the local model to the target model.|
|train             | train the agent during num_frames.                   |
|test              | test the agent (1 episode).                          |
|plot              | plot the training progresses.                        |

In the paper, NoisyNet is used as a component of the Dueling Network Architecture, which includes Double-DQN and Prioritized Experience Replay. However, we don't implement them to simplify the tutorial. One thing to note is that NoisyNet is an alternertive to $\epsilon$-greedy method, so all $\epsilon$ related lines are removed. Please check all comments with *NoisyNet*.

```python
class DQNAgent:
    """DQN Agent interacting with environment.

    Attribute:
        env (gym.Env): openAI Gym environment
        memory (ReplayBuffer): replay memory to store transitions
        batch_size (int): batch size for sampling
        target_update (int): period for target model's hard update
        gamma (float): discount factor
        dqn (Network): model to train and select actions
        dqn_target (Network): target model to update
        optimizer (torch.optim): optimizer for training dqn
        transition (list): transition information including
                           state, action, reward, next_state, done
    """

    def __init__(
        self,
        env: gym.Env,
        memory_size: int,
        batch_size: int,
        target_update: int,
        gamma: float = 0.99,
    ):
        """Initialization.

        Args:
            env (gym.Env): openAI Gym environment
            memory_size (int): length of memory
            batch_size (int): batch size for sampling
            target_update (int): period for target model's hard update
            gamma (float): discount factor
        """
        # NoisyNet: All attributes related to epsilon are removed
        obs_dim = env.observation_space.shape[0]
        action_dim = env.action_space.n

        self.env = env
        self.memory = ReplayBuffer(obs_dim, memory_size, batch_size)
        self.batch_size = batch_size
        self.target_update = target_update
        self.gamma = gamma

        # device: cpu / gpu
        self.device = torch.device(
            "cuda" if torch.cuda.is_available() else "cpu"
        )
        print(self.device)

        # networks: dqn, dqn_target
        self.dqn = Network(obs_dim, action_dim).to(self.device)
        self.dqn_target = Network(obs_dim, action_dim).to(self.device)
        self.dqn_target.load_state_dict(self.dqn.state_dict())
        self.dqn_target.eval()

        # optimizer
        self.optimizer = optim.Adam(self.dqn.parameters())

        # transition to store in memory
        self.transition = list()

        # mode: train / test
        self.is_test = False

    def select_action(self, state: np.ndarray) -> np.ndarray:
        """Select an action from the input state."""
        # NoisyNet: no epsilon greedy action selection
        selected_action = self.dqn(
            torch.FloatTensor(state).to(self.device)
        ).argmax()
        selected_action = selected_action.detach().cpu().numpy()

        if not self.is_test:
            self.transition = [state, selected_action]

        return selected_action

    def step(self, action: np.ndarray) -> Tuple[np.ndarray, np.float64, bool]:
        """Take an action and return the response of the env."""
        next_state, reward, done, _ = self.env.step(action)

        if not self.is_test:
            self.transition += [reward, next_state, done]
            self.memory.store(*self.transition)

        return next_state, reward, done

    def update_model(self) -> torch.Tensor:
        """Update the model by gradient descent."""
        samples = self.memory.sample_batch()

        loss = self._compute_dqn_loss(samples)

        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

        # NoisyNet: reset noise
        self.dqn.reset_noise()
        self.dqn_target.reset_noise()

        return loss.item()

    def train(self, num_frames: int, plotting_interval: int = 200):
        """Train the agent."""
        self.is_test = False

        state = self.env.reset()
        update_cnt = 0
        losses = []
        scores = []
        score = 0

        for frame_idx in range(1, num_frames + 1):
            action = self.select_action(state)
            next_state, reward, done = self.step(action)

            state = next_state
            score += reward

            # NoisyNet: removed decrease of epsilon

            # if episode ends
            if done:
                state = env.reset()
                scores.append(score)
                score = 0

            # if training is ready
            if len(self.memory) >= self.batch_size:
                loss = self.update_model()
                losses.append(loss)
                update_cnt += 1

                # if hard update is needed
                if update_cnt % self.target_update == 0:
                    self._target_hard_update()

            # plotting
            if frame_idx % plotting_interval == 0:
                self._plot(frame_idx, scores, losses)

        self.env.close()

    def test(self) -> List[np.ndarray]:
        """Test the agent."""
        self.is_test = True

        state = self.env.reset()
        done = False
        score = 0

        frames = []
        while not done:
            frames.append(self.env.render(mode="rgb_array"))
            action = self.select_action(state)
            next_state, reward, done = self.step(action)

            state = next_state
            score += reward

        print("score: ", score)
        self.env.close()

        return frames

    def _compute_dqn_loss(self, samples: Dict[str, np.ndarray]) -> torch.Tensor:
        """Return dqn loss."""
        device = self.device  # for shortening the following lines
        state = torch.FloatTensor(samples["obs"]).to(device)
        next_state = torch.FloatTensor(samples["next_obs"]).to(device)
        action = torch.LongTensor(samples["acts"].reshape(-1, 1)).to(device)
        reward = torch.FloatTensor(samples["rews"].reshape(-1, 1)).to(device)
        done = torch.FloatTensor(samples["done"].reshape(-1, 1)).to(device)

        # G_t   = r + gamma * v(s_{t+1})  if state != Terminal
        #       = r                       otherwise
        curr_q_value = self.dqn(state).gather(1, action)
        next_q_value = self.dqn_target(next_state).max(
            dim=1, keepdim=True
        )[0].detach()
        mask = 1 - done
        target = (reward + self.gamma * next_q_value * mask).to(self.device)

        # calculate dqn loss
        loss = F.smooth_l1_loss(curr_q_value, target)

        return loss

    def _target_hard_update(self):
        """Hard update: target <- local."""
        self.dqn_target.load_state_dict(self.dqn.state_dict())

    def _plot(
        self,
        frame_idx: int,
        scores: List[float],
        losses: List[float],
    ):
        """Plot the training progresses."""
        clear_output(True)
        plt.figure(figsize=(20, 5))
        plt.subplot(131)
        plt.title('frame %s. score: %s' % (frame_idx, np.mean(scores[-10:])))
        plt.plot(scores)
        plt.subplot(132)
        plt.title('loss')
        plt.plot(losses)
        plt.show()
```

## Environment

You can see the [code](https://github.com/openai/gym/blob/master/gym/envs/classic_control/cartpole.py) and [configurations](https://github.com/openai/gym/blob/master/gym/envs/__init__.py#L53) of CartPole-v0 from OpenAI's repository.

```python
# environment
env_id = "CartPole-v0"
env = gym.make(env_id)
```

## Set random seed

```python
seed = 777

def seed_torch(seed):
    torch.manual_seed(seed)
    if torch.backends.cudnn.enabled:
        torch.backends.cudnn.benchmark = False
        torch.backends.cudnn.deterministic = True

np.random.seed(seed)
seed_torch(seed)
env.seed(seed)
```

    [777]

## Initialize

```python
# parameters
num_frames = 10000
memory_size = 1000
batch_size = 32
target_update = 100

# train
agent = DQNAgent(env, memory_size, batch_size, target_update)
```

    cuda

## Train

```python
agent.train(num_frames)
```

## Test

Run the trained agent (1 episode).

```python
frames = agent.test()
```

    score:  200.0

## Render

```python
# Imports specifically so we can render outputs in Colab.
from matplotlib import animation
from JSAnimation.IPython_display import display_animation
from IPython.display import display

def display_frames_as_gif(frames):
    """Displays a list of frames as a gif, with controls."""
    patch = plt.imshow(frames[0])
    plt.axis('off')

    def animate(i):
        patch.set_data(frames[i])

    anim = animation.FuncAnimation(
        plt.gcf(), animate, frames = len(frames), interval=50
    )
    display(display_animation(anim, default_mode='loop'))

# display
display_frames_as_gif(frames)
```

    <input id="_anim_sliderDVREKFGCMTZZTQWF" type="range" style="width:350px" name="points" min="0" max="1" step="1" value="0" onchange="animDVREKFGCMTZZTQWF.set_frame(parseInt(this.value));"></input>

    <button onclick="animDVREKFGCMTZZTQWF.slower()">–</button>
    <button onclick="animDVREKFGCMTZZTQWF.first_frame()"></button>
    <button onclick="animDVREKFGCMTZZTQWF.previous_frame()"></button>
    <button onclick="animDVREKFGCMTZZTQWF.reverse_animation()"></button>
    <button onclick="animDVREKFGCMTZZTQWF.pause_animation()"></button>
    <button onclick="animDVREKFGCMTZZTQWF.play_animation()"></button>
    <button onclick="animDVREKFGCMTZZTQWF.next_frame()"></button>
    <button onclick="animDVREKFGCMTZZTQWF.last_frame()"></button>
    <button onclick="animDVREKFGCMTZZTQWF.faster()">+</button>
  <form action="#n" name="_anim_loop_selectDVREKFGCMTZZTQWF" class="anim_control">
    <input type="radio" name="state" value="once" > Once </input>
    <input type="radio" name="state" value="loop" checked> Loop </input>
    <input type="radio" name="state" value="reflect" > Reflect </input>
  </form>