<a href="https://colab.research.google.com/github/nickwotton/MQP2019/blob/master/Nick/NN_linearfunction01.ipynb" target="_parent"></a>

# Attempt to Improve Solving a Linear Function using a Neural Network
Given code to use a neural network to fit a linear function, try to optimize the code to get a better fit, i.e. the data points completely overlap on the plot.


```python
import torch
import torch.nn as nn
import numpy as np
import torch.nn.functional as F
import matplotlib.pyplot as plt
```

## Define the Function
Here we define our function $f(x)=ax+b$ with coefficient $a=1$ and intercept term $b=2$.
Then we test the equation with a test value of 2.


```python
# target function
a = 1.
b = 2.
f = lambda x: a*x+b

#test
f(2.)
```




    4.0



## Create Model
Next, we create the neural network model. This is done first by setting the inner and outer dimensions with variables. Next we code the model and vary the internal dimensions to attempt to improve the model. At this level, this is essentially a simple linear algebra exercise:

If we have input $x$, internal parameters $a,b$, and solution $f(x)$ then in the one-dimensional case we have:

\begin{equation}
  \left(
    a_{1}x+b_{2}
  \right)
  a_{2} + b_{2}
  = f(x)
\end{equation}
  
However, we want to get a better estimate for the true equation. So we increase the interior dimension which corresponds to the number of neurons inside the network. For example, we raised the inner dimension to 3. In matrix form we have:

\begin{equation}
\left(
  \begin{bmatrix} x \end{bmatrix} 
  \begin{bmatrix} a_{1} & a_{2} & a_{3} \end{bmatrix} 
  + 
  \begin{bmatrix} b_{1} & b_{2} & b_{3} \end{bmatrix}
\right)
 \begin{bmatrix} a_{4} \\ a_{5} \\ a_{6} \end{bmatrix}
 +
 \begin{bmatrix} b_{4} \\ \end{bmatrix}
 =
 \begin{bmatrix} f(x) \end{bmatrix}
\end{equation}

Graphically, we can render this second neural network as:


What we discovered here is that ReLU was slowing down the process, so since our function is Linear, we can just remove it. 

Additionally, we discerned that the higher the inner dimension, that is, the more nodes in each layer, the smaller the error and the better the performance.


```python
#model
#nn.Linear
in_dim = 1
out_dim = 1

model = nn.Sequential(
    nn.Linear(in_dim, 30),
#    nn.ReLU(),
    nn.Linear(30, out_dim)
)
```

Here we define the Loss function as the Mean Squared Error(MSE). 

Note that by doing so, we are essentially 'cheating' the system. In most applications, we would not know the function $f$ so we would be unable to find the MSE.


```python
#loss function 
criterion = nn.MSELoss()
```

Next we choose a learning rate and a method for learning. The learning rate is the percent of the data that is accepted in each iteration. The Methods we tried were SGD and Adam. 


```python
#optimizer
learning_rate = 0.001
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate) 
```

## Train the Model

First we create the training data. This is a batch of random points that we pass through our function $f$.


```python
#training data

batch_size = 1000

x_train = torch.randn(batch_size, 1)
y_train = f(x_train)

```

Once we have the training data, we pass this collection of inputs and solutions into the model. With each iteration we calculate the loss and attempt to optimize the model to further reduce the loss.

In this code we print out the loss every 10 iterations.


```python
# Train the model

num_epochs = 500

for epoch in range(num_epochs):

    # Forward pass
    outputs = model(x_train)
    loss = criterion(outputs, y_train)
    
    # Backward and optimize
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    
    if (epoch+1) % 10 == 0:
        print ('Epoch [{}/{}], Loss: {:.4f}'.format(epoch+1, 
                                                    num_epochs, loss.item()))

```

    Epoch [10/500], Loss: 3.6593
    Epoch [20/500], Loss: 2.1498
    Epoch [30/500], Loss: 1.2767
    Epoch [40/500], Loss: 0.7678
    Epoch [50/500], Loss: 0.4685
    Epoch [60/500], Loss: 0.2906
    Epoch [70/500], Loss: 0.1835
    Epoch [80/500], Loss: 0.1180
    Epoch [90/500], Loss: 0.0773
    Epoch [100/500], Loss: 0.0515
    Epoch [110/500], Loss: 0.0348
    Epoch [120/500], Loss: 0.0239
    Epoch [130/500], Loss: 0.0166
    Epoch [140/500], Loss: 0.0116
    Epoch [150/500], Loss: 0.0082
    Epoch [160/500], Loss: 0.0058
    Epoch [170/500], Loss: 0.0041
    Epoch [180/500], Loss: 0.0030
    Epoch [190/500], Loss: 0.0021
    Epoch [200/500], Loss: 0.0015
    Epoch [210/500], Loss: 0.0011
    Epoch [220/500], Loss: 0.0008
    Epoch [230/500], Loss: 0.0006
    Epoch [240/500], Loss: 0.0004
    Epoch [250/500], Loss: 0.0003
    Epoch [260/500], Loss: 0.0002
    Epoch [270/500], Loss: 0.0002
    Epoch [280/500], Loss: 0.0001
    Epoch [290/500], Loss: 0.0001
    Epoch [300/500], Loss: 0.0001
    Epoch [310/500], Loss: 0.0000
    Epoch [320/500], Loss: 0.0000
    Epoch [330/500], Loss: 0.0000
    Epoch [340/500], Loss: 0.0000
    Epoch [350/500], Loss: 0.0000
    Epoch [360/500], Loss: 0.0000
    Epoch [370/500], Loss: 0.0000
    Epoch [380/500], Loss: 0.0000
    Epoch [390/500], Loss: 0.0000
    Epoch [400/500], Loss: 0.0000
    Epoch [410/500], Loss: 0.0000
    Epoch [420/500], Loss: 0.0000
    Epoch [430/500], Loss: 0.0000
    Epoch [440/500], Loss: 0.0000
    Epoch [450/500], Loss: 0.0000
    Epoch [460/500], Loss: 0.0000
    Epoch [470/500], Loss: 0.0000
    Epoch [480/500], Loss: 0.0000
    Epoch [490/500], Loss: 0.0000
    Epoch [500/500], Loss: 0.0000


## Testing the Model
Now that we have a trained model with low Loss, we want to attempt to replicate the function. To do this we get another random sample of numbers. This sample is passed into both our fuction $f$ and the model.

We then graph both sets of points on a scatter plot. Since the model is highly accurate now, the two sets of points completely overlap.


```python
#test
x_ = torch.randn(50,1)
y_ = f(x_)
plt.scatter(x_.detach().numpy(), y_.detach().numpy(), label='true')

y_pred = model(x_)
plt.scatter(x_.detach().numpy(), y_pred.detach().numpy(), label='pred')

plt.legend()
```

368c8844a3be12957deefdd2853a917567f1779f
ipynb
Jupyter Notebook
Nick/NN_linearfunction01.ipynb
vitaltavares/MQP2019
d7bda131450907beef3c9d619acf36d6c1ced2a9
MIT
2019-09-13T16:27:14.000Z
2019-09-13T16:27:14.000Z
Nick/NN_linearfunction01.ipynb
vitaltavares/MQP2019
d7bda131450907beef3c9d619acf36d6c1ced2a9
MIT
2019-09-02T17:49:28.000Z
2020-04-06T13:03:43.000Z
Nick/NN_linearfunction01.ipynb
vitaltavares/MQP2019
d7bda131450907beef3c9d619acf36d6c1ced2a9
MIT
2019-09-11T01:51:56.000Z
2019-12-03T20:13:16.000Z
Qwen/Qwen-72B
1. YES
2. YES