# CHAPTER 5

# Section 1

Learning about how to fit generic mathematical functions in PyTorch, not just for neural networks.



```python
import torch
```


```python
# trying to calibrate a new thermometer
```


```python
truth_celsius = [0.5, 14.0, 15.0, 28.0, 11.0, 8.0, 3.0, -4.0, 6.0, 13.0, 21.0] # celsius
x = [35.7, 55.9, 58.2, 81.9, 56.3, 48.9, 33.9, 21.8, 48.4, 60.4, 68.4] # unknown shit
```


```python
truth_celsius = torch.tensor(truth_celsius)
x = torch.tensor(x)

type(truth_celsius), type(x)
```




    (torch.Tensor, torch.Tensor)




```python
# testing a linear model

# celsius = w * x + b
```


```python
import matplotlib.pyplot as plt

plt.scatter(x, truth_celsius)
```


```python
# a simple model

def model(data, w, b):
    return data * w + b
```


```python
# loss function - mean squared difference is used because it is better to have a few slightly
# higher losses than a few really high ones, which mean squared error takes care of

def loss(y, y_pred):
    return torch.mean(torch.pow(y - y_pred, 2))
```


```python
# initialize params

# we don't need to specify the shape of these params
# because while using multiplication with pytorch tensors, they will be broadcasted
w = torch.ones(())
b = torch.zeros(())
```


```python
# forward pass

preds = model(x, w, b)
preds
```




    tensor([35.7000, 55.9000, 58.2000, 81.9000, 56.3000, 48.9000, 33.9000, 21.8000,
            48.4000, 60.4000, 68.4000])




```python
loss_val = loss(preds, truth_celsius)
loss_val
```




    tensor(1763.8846)



The hypothesis (forward pass) function is defined as:

$$
\large h(x) = {w \cdot x + b}
$$

To generalize the hypothesis to arbritrary parameters and inputs in a univariate linear regression, we derive:

$$
\large h_{\theta}(x) = \theta_0 x_0 + \theta_1 x_1
$$

We will assume the following:

$$
\large b = \theta_0 \\
\large w = \theta_1 \\
$$

Since $b$ or $\theta_0$ is a constant/intercept, it will be multiplied with $1$.

$$
\large x_0 = 1
$$

Hence the loss function will be:

$$
\large \hat{L}_{(y, \hat{y})} = \frac{1}{2m} \sum_{i = 1}^{m}\, (\hat{y}^{(i)} - y^{(i)}) ^ 2 \qquad ,y^{(i)} = h_{\theta}(x^{(i)})
$$

Substituting the hypothesis function into the loss function:

$$
\large \hat{L}_{(\theta_0, \theta_1)} = \frac{1}{2m} \sum_{i = 1}^{m}\, (h_{\theta}(x^{(i)}) - y^{(i)}) ^ 2 \qquad ,h_{\theta}(x) = \theta_0 x_0 + \theta_1 x_1
$$

__Note:__ The _2_ in the denominator of the loss function is placed there for convenience so that when the derivative of the loss function is taken, the _2's_ divide out and we are left with $\large\frac{1}{m}$
however this is not implemented in the code because for optimization purposes it is unnecessary.


The $\large\frac{1}{m}$ is in the equation so that the loss function does not depend on the number of examples in the dataset, and its values are comparable across any dataset size.

---

The derivative of the loss function with respect to its parameters according to the Chain rule will be:

$$
\large \frac{\partial}{\partial \theta_{0, 1}}\hat{L}(\theta_0, \theta_1) = 
\frac{\partial \hat{L}(\theta_0, \theta_1)}{\partial h_{\theta}(x)} \times
\frac{\partial h_{\theta}(x)}{\partial \theta_{0, 1}}
$$

The derivative of the loss function with respect to the hypothesis function is as follows:

$$
\large\frac{\partial \hat{L}(\theta_0, \theta_1)}{\partial h_{\theta}(x)} = 
\frac{1}{m} \sum_{i = 1}^{m}\, (h_{\theta}(x^{(i)}) - y^{(i)}) \qquad , h_{\theta}(x) = \theta_0 x_0 + \theta_1 x_1 
$$

---

The derivative of the hypothesis function with respect to $\theta_0$ will be:

$$
\large\begin{equation}
\frac{\partial}{\partial \theta_0} h_{\theta}(x) = \frac{\partial}{\partial \theta_0} (\theta_0 x_0 + \theta_1 x_1) \\
\frac{\partial}{\partial \theta_0} h_{\theta}(x) = \frac{\partial}{\partial \theta_0} (\theta_0 x_0) + \frac{\partial}{\partial \theta_0} (\theta_1 x_1) \\
\frac{\partial}{\partial \theta_0} h_{\theta}(x) = \frac{\partial}{\partial \theta_0} (\theta_0 x_0) + \textit{0} \\
\frac{\partial}{\partial \theta_0} h_{\theta}(x) = x_0 = 1
\end{equation}
$$

The value of $\large\frac{\partial}{\partial \theta_0} h_{\theta}(x)$ will be broadcasted to a vector so that it is compatible to be multiplied with $\large\frac{\partial \hat{L}(\theta_0, \theta_1)}{\partial h_{\theta}(x)}$.

---

The derivative of the hypothesis function with respect to $\large\theta_1$ will be:

$$
\large\begin{equation}
\frac{\partial}{\partial \theta_1} h_{\theta}(x) = \frac{\partial}{\partial \theta_1} (\theta_0 x_0 + \theta_1 x_1) \\
\frac{\partial}{\partial \theta_1} h_{\theta}(x) = \frac{\partial}{\partial \theta_1} (\theta_0 x_0) + \frac{\partial}{\partial \theta_1} (\theta_1 x_1) \\
\frac{\partial}{\partial \theta_1} h_{\theta}(x) = \textit{0} + \frac{\partial}{\partial \theta_1} (\theta_1 x_1) \\
\frac{\partial}{\partial \theta_1} h_{\theta}(x) = x_1 \\
x_1 = \vec{x_1}
\end{equation}
$$

$\large\vec{x_1}$ represents the input vector to the model (the list of temperature values in this case).

---

The gradient of the loss function will therefore be:

$$
\Large\nabla_{\theta_0, \theta_1} \hat{L} = 
\begin{pmatrix}
\frac{\partial \hat{L}}{\partial \theta_0}, \frac{\partial \hat{L}}{\partial \theta_1}
\end{pmatrix}
=
\begin{pmatrix}
\frac{\partial \hat{L}}{\partial h_{\theta}(x)} \frac{\partial h_{\theta}(x)}{\partial \theta_0},
\frac{\partial \hat{L}}{\partial h_{\theta}(x)} \frac{\partial h_{\theta}(x)}{\partial \theta_1}
\end{pmatrix}
$$


```python
def dl_h(preds, truth):
    derivative = 2 * (preds - truth) / preds.size(0)
    return derivative
```


```python
dh_w = lambda data, w, b: data
```


```python
dh_b = lambda data, w, b: 1.0 # this will be broadcasted into a vector when multiplied with dl_dh
```


```python
# a function to get all the gradients together

def grad(data, preds, truth, w, b):
    l = loss(preds, truth)
    
    dl_dh = dl_h(preds, truth) # this is an intermediate derivative. It's not of much use
    
    # these 2 statements result in vectors
    dl_dw = dl_dh * dh_w(data, w, b)
    dl_db = dl_dh * dh_b(data, w, b)
    
    return torch.stack([dl_dw.sum(), dl_db.sum()]) # sum is done to make the vectors into a scalar
```

### Training


```python
def train_loop(epochs, lr, params, data, truth):
    for epoch in range(1, epochs+1):
        
        w, b = params
        
        curr_pred = model(data, w, b)
        
        curr_loss = loss(curr_pred, truth)
        
        gradient = grad(data, curr_pred, truth, w, b)
        
        params -= lr * gradient
        
        if epochs > 1000:
            if epoch % 100 == 0:
                print(f"Epoch {epoch}:\tloss = {torch.round(curr_loss)}\ngrad: dw: {torch.round(gradient[0])} db: {torch.round(gradient[1])}\nparams: w: {params[0]} b: {params[1]}\n")
        else:
            if epoch % 10 == 0:
                print(f"Epoch {epoch}:\tloss = {torch.round(curr_loss)}\ngrad: dw: {torch.round(gradient[0])} db: {torch.round(gradient[1])}\nparams: w: {params[0]} b: {params[1]}\n")
```


```python
truth_celsius, x
```




    (tensor([ 0.5000, 14.0000, 15.0000, 28.0000, 11.0000,  8.0000,  3.0000, -4.0000,
              6.0000, 13.0000, 21.0000]),
     tensor([35.7000, 55.9000, 58.2000, 81.9000, 56.3000, 48.9000, 33.9000, 21.8000,
             48.4000, 60.4000, 68.4000]))




```python
# WARNING: BRACE FOR FAILURE
train_loop(40, 0.001, torch.tensor([1.0, 0.0]), x, truth_celsius)
```

    Epoch 10:	loss = 4327910768902144.0
    grad: dw: -7135127040.0 db: -125683768.0
    params: w: 5922376.0 b: 104321.3671875
    
    Epoch 20:	loss = 2.574914148345635e+29
    grad: dw: -5.503563362153267e+16 db: -969441227571200.0
    params: w: 45681276354560.0 b: 804666212352.0
    
    Epoch 30:	loss = inf
    grad: dw: -4.245086883863936e+23 db: -7.477631017248035e+21
    params: w: 3.523553368428637e+20 b: 6.206665073469948e+18
    
    Epoch 40:	loss = inf
    grad: dw: -3.274379209203682e+30 db: -5.767750612106071e+28
    params: w: 2.717836133130785e+27 b: 4.7874115471479525e+25
    


#### See kids, this is why you don't set the learning rate to be too high. The training blows up.


```python
train_loop(40, 1e-4, torch.tensor([1.0, 0.0]), x, truth_celsius)
```

    Epoch 10:	loss = 29.0
    grad: dw: 1.0 db: 3.0
    params: w: 0.23235063254833221 b: -0.016550514847040176
    
    Epoch 20:	loss = 29.0
    grad: dw: -0.0 db: 3.0
    params: w: 0.2322966605424881 b: -0.01957947015762329
    
    Epoch 30:	loss = 29.0
    grad: dw: -0.0 db: 3.0
    params: w: 0.2323499619960785 b: -0.022606005892157555
    
    Epoch 40:	loss = 29.0
    grad: dw: -0.0 db: 3.0
    params: w: 0.23240326344966888 b: -0.025632008910179138
    


#### Now kids, this is what we call converging.

#### Let's try input normalization for the input vector so that the gradients (w and b) are not different from each other. This will allow the learning rate to meaningfully update both parameters.


```python
x_n = 0.1 * x
x_n, truth_celsius
```




    (tensor([3.5700, 5.5900, 5.8200, 8.1900, 5.6300, 4.8900, 3.3900, 2.1800, 4.8400,
             6.0400, 6.8400]),
     tensor([ 0.5000, 14.0000, 15.0000, 28.0000, 11.0000,  8.0000,  3.0000, -4.0000,
              6.0000, 13.0000, 21.0000]))




```python
train_loop(100, 1e-2, torch.tensor([1.0, 0.0]), x_n, truth_celsius)
```

    Epoch 10:	loss = 29.0
    grad: dw: -1.0 db: 3.0
    params: w: 2.3232028484344482 b: -0.07098990678787231
    
    Epoch 20:	loss = 28.0
    grad: dw: -1.0 db: 3.0
    params: w: 2.3746397495269775 b: -0.361503541469574
    
    Epoch 30:	loss = 27.0
    grad: dw: -1.0 db: 3.0
    params: w: 2.425097942352295 b: -0.6471392512321472
    
    Epoch 40:	loss = 26.0
    grad: dw: -0.0 db: 3.0
    params: w: 2.474705696105957 b: -0.9279596209526062
    
    Epoch 50:	loss = 26.0
    grad: dw: -0.0 db: 3.0
    params: w: 2.523477077484131 b: -1.2040457725524902
    
    Epoch 60:	loss = 25.0
    grad: dw: -0.0 db: 3.0
    params: w: 2.5714263916015625 b: -1.4754774570465088
    
    Epoch 70:	loss = 24.0
    grad: dw: -0.0 db: 3.0
    params: w: 2.6185672283172607 b: -1.7423334121704102
    
    Epoch 80:	loss = 24.0
    grad: dw: -0.0 db: 3.0
    params: w: 2.6649134159088135 b: -2.004690647125244
    
    Epoch 90:	loss = 23.0
    grad: dw: -0.0 db: 3.0
    params: w: 2.7104780673980713 b: -2.262624740600586
    
    Epoch 100:	loss = 22.0
    grad: dw: -0.0 db: 3.0
    params: w: 2.755274772644043 b: -2.5162105560302734
    



```python
train_loop(5000, 1e-2, torch.tensor([1.0, 0.0]), x_n, truth_celsius)
```

    Epoch 100:	loss = 22.0
    grad: dw: -0.0 db: 3.0
    params: w: 2.755274772644043 b: -2.5162105560302734
    
    Epoch 200:	loss = 17.0
    grad: dw: -0.0 db: 2.0
    params: w: 3.1637372970581055 b: -4.8284430503845215
    
    Epoch 300:	loss = 13.0
    grad: dw: -0.0 db: 2.0
    params: w: 3.508335590362549 b: -6.77915096282959
    
    Epoch 400:	loss = 10.0
    grad: dw: -0.0 db: 2.0
    params: w: 3.799055337905884 b: -8.424863815307617
    
    Epoch 500:	loss = 8.0
    grad: dw: -0.0 db: 1.0
    params: w: 4.044321060180664 b: -9.813268661499023
    
    Epoch 600:	loss = 6.0
    grad: dw: -0.0 db: 1.0
    params: w: 4.251237869262695 b: -10.984587669372559
    
    Epoch 700:	loss = 5.0
    grad: dw: -0.0 db: 1.0
    params: w: 4.425803184509277 b: -11.972770690917969
    
    Epoch 800:	loss = 5.0
    grad: dw: -0.0 db: 1.0
    params: w: 4.573074817657471 b: -12.806446075439453
    
    Epoch 900:	loss = 4.0
    grad: dw: -0.0 db: 1.0
    params: w: 4.697319507598877 b: -13.509775161743164
    
    Epoch 1000:	loss = 4.0
    grad: dw: -0.0 db: 1.0
    params: w: 4.802137851715088 b: -14.103133201599121
    
    Epoch 1100:	loss = 4.0
    grad: dw: -0.0 db: 0.0
    params: w: 4.890567779541016 b: -14.603717803955078
    
    Epoch 1200:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 4.9651713371276855 b: -15.02603530883789
    
    Epoch 1300:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.028110504150391 b: -15.382323265075684
    
    Epoch 1400:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.081210136413574 b: -15.682910919189453
    
    Epoch 1500:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.126006126403809 b: -15.936492919921875
    
    Epoch 1600:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.163797855377197 b: -16.15042495727539
    
    Epoch 1700:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.195681571960449 b: -16.33091163635254
    
    Epoch 1800:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.222580432891846 b: -16.48318099975586
    
    Epoch 1900:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.245273113250732 b: -16.61164093017578
    
    Epoch 2000:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.264420032501221 b: -16.720029830932617
    
    Epoch 2100:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.280571460723877 b: -16.811458587646484
    
    Epoch 2200:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.294197082519531 b: -16.88858985900879
    
    Epoch 2300:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.305691719055176 b: -16.95366096496582
    
    Epoch 2400:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.315390110015869 b: -17.008562088012695
    
    Epoch 2500:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.32357120513916 b: -17.054872512817383
    
    Epoch 2600:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.330473899841309 b: -17.093948364257812
    
    Epoch 2700:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.336297035217285 b: -17.12691307067871
    
    Epoch 2800:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.34121036529541 b: -17.15472412109375
    
    Epoch 2900:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.345354080200195 b: -17.178180694580078
    
    Epoch 3000:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.348850727081299 b: -17.197975158691406
    
    Epoch 3100:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.351799964904785 b: -17.214672088623047
    
    Epoch 3200:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.354288578033447 b: -17.228757858276367
    
    Epoch 3300:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.356388092041016 b: -17.240642547607422
    
    Epoch 3400:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.35815954208374 b: -17.250669479370117
    
    Epoch 3500:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.359652996063232 b: -17.259126663208008
    
    Epoch 3600:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.3609137535095215 b: -17.26626205444336
    
    Epoch 3700:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.3619771003723145 b: -17.272281646728516
    
    Epoch 3800:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.362875461578369 b: -17.277366638183594
    
    Epoch 3900:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.3636322021484375 b: -17.28165054321289
    
    Epoch 4000:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.364270210266113 b: -17.285263061523438
    
    Epoch 4100:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.364808559417725 b: -17.288311004638672
    
    Epoch 4200:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.36526346206665 b: -17.290884017944336
    
    Epoch 4300:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.365647315979004 b: -17.293058395385742
    
    Epoch 4400:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.365969181060791 b: -17.294879913330078
    
    Epoch 4500:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.366244316101074 b: -17.296436309814453
    
    Epoch 4600:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.366473197937012 b: -17.297733306884766
    
    Epoch 4700:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.3666672706604 b: -17.298831939697266
    
    Epoch 4800:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.366833686828613 b: -17.299772262573242
    
    Epoch 4900:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.366968154907227 b: -17.300535202026367
    
    Epoch 5000:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.367083549499512 b: -17.301189422607422
    



```python
param = (5.367083549499512, -17.301189422607422)
preds = model(x_n, *param)
fig = plt.figure(dpi=600)
plt.xlabel("Temperature (°Fahrenheit)")
plt.ylabel("Temperature (°Celsius)")
plt.plot(x.numpy(), preds.detach().numpy())
plt.plot(x.numpy(), truth_celsius.numpy(), 'o')
```

### Computing the gradient automatically with PyTorch

### So PyTorch has something called __autograd__ apparently. A property of its tensors that gives them the ability to remember where they come from i.e. their parentage and origin of tensors. Therefore, any nested graph of tensors will automatically be differentiated with autograd, no matter how complex.


```python
# we already have the loss function and the hypothesis function

model?
```


    [0;31mSignature:[0m [0mmodel[0m[0;34m([0m[0mdata[0m[0;34m,[0m [0mw[0m[0;34m,[0m [0mb[0m[0;34m)[0m[0;34m[0m[0m
    [0;31mDocstring:[0m <no docstring>
    [0;31mFile:[0m      ~/pytorch_practice/Chapter_5/<ipython-input-7-a61dd8821a74>
    [0;31mType:[0m      function




```python
loss?
```


    [0;31mSignature:[0m [0mloss[0m[0;34m([0m[0my[0m[0;34m,[0m [0my_pred[0m[0;34m)[0m[0;34m[0m[0m
    [0;31mDocstring:[0m <no docstring>
    [0;31mFile:[0m      ~/pytorch_practice/Chapter_5/<ipython-input-8-e2d7d2cae26a>
    [0;31mType:[0m      function




```python
# initializing the parameters tensor

params = torch.tensor([1.0, 0.0], requires_grad=True)
# the requires_grad parameter tracks all the family tree folks that operate on params. Any tensor that has params as an 
# ancestor will have access to the sequence of functions it took from params to get to that tensor.

# If those tensor functions/operations are differentiable (as most are), they will be added to the grad attribute in params

params
```




    tensor([1., 0.], requires_grad=True)




```python
params.grad is None
```




    True




```python
l = loss(model(x, *params), truth_celsius)

l.backward() # this will compute the derivative of loss with respect to the elements in params
```


```python
params.grad
```




    tensor([4517.2969,   82.6000])



# __NOTE: Calling backward will lead derivatives to accumulate at leaf nodes. We need to zero the gradient explicitly after using it for parameter updates. So if backward was called earlier, the loss is evaluated again, backward is called again (as in any training loop), and the gradient at each leaf is accumulated (that is, summed) on top of the one computed at the previous iteration, which leads to an incorrect value for the gradient.__


```python
# which is why after each iteration, the gradients must be zeroed

params.grad.zero_()
params.grad
```




    tensor([0., 0.])



### Experimentation with autograd


```python
d = torch.tensor([4], dtype=torch.float, requires_grad=True) # by the way, only floating point tensors can have requires_grad=True
f = d ** 2

f
```




    tensor([16.], grad_fn=<PowBackward0>)




```python
f.backward()
```


```python
print(d.grad)

d.grad.zero_();
```

    tensor([8.])


$$
\large\begin{equation}
f(d) = d^2 \\
f'(d) = 2 d
\end{equation}
$$


```python
# let's try one composite function

d = torch.tensor([2], dtype=torch.float, requires_grad=True)

f = d ** 3
t = d ** 2

g = f * t
```

$$
\large\begin{equation}
f(d) = d^3 \\
t(d) = d^2 \\
g(d) = f(d) \cdot t(d) \\
\end{equation}
$$

When we differentiate $\large g(d)$ with respect to $\large d$, we get

$$
\large\begin{equation}
g'(d) = f'(d) \cdot t(d) + f(d) \cdot t'(d) \\
g'(d) = 3 d^2 \cdot d^2 + d^3 \cdot 2 d \\
g'(d) = 3 d^4 + 2 d^4 \\
g'(d) = 5 d^4
\end{equation}
$$


```python
# to compute this derivative, we do

g.backward()
```


```python
print(d.grad)

d.grad.zero_();

# 80 is correct because 5 * 2^4 = 80
```

    tensor([80.])



```python
def train_loop(epochs, lr, params, data, truth):
    for epoch in range(1, epochs+1):
        
        w, b = params
        
        curr_pred = model(data, w, b)
        
        curr_loss = loss(curr_pred, truth)
        
        curr_loss.backward() # computing the derivatives
        
        gradient = params.grad
        
        # THIS TELLS THE AUTOGRAD TO LOOK AWAY WHEN RUNNING THE CODE
        # THAT IS NOT ADD EDGES TO THE FORWARD GRAPH
        with torch.no_grad(): # THIS CODE IS REQUIRED WHEN DOING THE PARAMETER UPDATE
            params -= lr * gradient # this is an example of element wise multiplication and subtraction
            
        # WHEN TORCH.NO_GRAD() IS CALLED, THE FORWARD GRAPH IS CONSUMED, THE PARAMS LEAF NODE IS UPDATED
        # BEFORE MAKING A FRESH FORWARD GRAPH IN THE NEXT LOSS.BACKWARD() CALL
        
        if epochs > 1000:
            if epoch % 100 == 0:
                print(f"Epoch {epoch}:\tloss = {torch.round(curr_loss)}\ngrad: dw: {torch.round(gradient[0])} db: {torch.round(gradient[1])}\nparams: w: {params[0]} b: {params[1]}\n")
        else:
            if epoch % 10 == 0:
                print(f"Epoch {epoch}:\tloss = {torch.round(curr_loss)}\ngrad: dw: {torch.round(gradient[0])} db: {torch.round(gradient[1])}\nparams: w: {params[0]} b: {params[1]}\n")
            
        params.grad.zero_() # this needs to be called before the next time loss.backward() is called so that the gradients do not accumulate

```


```python
params = torch.tensor([1.0, 0.0], requires_grad=True)

train_loop(5000, 1e-2, params, x_n, truth_celsius)
```

    Epoch 100:	loss = 22.0
    grad: dw: -0.0 db: 3.0
    params: w: 2.755274772644043 b: -2.5162105560302734
    
    Epoch 200:	loss = 17.0
    grad: dw: -0.0 db: 2.0
    params: w: 3.1637372970581055 b: -4.8284430503845215
    
    Epoch 300:	loss = 13.0
    grad: dw: -0.0 db: 2.0
    params: w: 3.508335590362549 b: -6.77915096282959
    
    Epoch 400:	loss = 10.0
    grad: dw: -0.0 db: 2.0
    params: w: 3.799055337905884 b: -8.424863815307617
    
    Epoch 500:	loss = 8.0
    grad: dw: -0.0 db: 1.0
    params: w: 4.044321060180664 b: -9.813268661499023
    
    Epoch 600:	loss = 6.0
    grad: dw: -0.0 db: 1.0
    params: w: 4.251237869262695 b: -10.984587669372559
    
    Epoch 700:	loss = 5.0
    grad: dw: -0.0 db: 1.0
    params: w: 4.425803184509277 b: -11.972770690917969
    
    Epoch 800:	loss = 5.0
    grad: dw: -0.0 db: 1.0
    params: w: 4.573074817657471 b: -12.806446075439453
    
    Epoch 900:	loss = 4.0
    grad: dw: -0.0 db: 1.0
    params: w: 4.697319507598877 b: -13.509775161743164
    
    Epoch 1000:	loss = 4.0
    grad: dw: -0.0 db: 1.0
    params: w: 4.802137851715088 b: -14.103133201599121
    
    Epoch 1100:	loss = 4.0
    grad: dw: -0.0 db: 0.0
    params: w: 4.890567779541016 b: -14.603717803955078
    
    Epoch 1200:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 4.9651713371276855 b: -15.02603530883789
    
    Epoch 1300:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.028110504150391 b: -15.382323265075684
    
    Epoch 1400:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.081210136413574 b: -15.682910919189453
    
    Epoch 1500:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.126006126403809 b: -15.936492919921875
    
    Epoch 1600:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.163797855377197 b: -16.15042495727539
    
    Epoch 1700:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.195681571960449 b: -16.33091163635254
    
    Epoch 1800:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.222580432891846 b: -16.48318099975586
    
    Epoch 1900:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.245273113250732 b: -16.61164093017578
    
    Epoch 2000:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.264420032501221 b: -16.720029830932617
    
    Epoch 2100:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.280571460723877 b: -16.811458587646484
    
    Epoch 2200:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.294197082519531 b: -16.88858985900879
    
    Epoch 2300:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.305691719055176 b: -16.95366096496582
    
    Epoch 2400:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.315390110015869 b: -17.008562088012695
    
    Epoch 2500:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.32357120513916 b: -17.054872512817383
    
    Epoch 2600:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.330473899841309 b: -17.093948364257812
    
    Epoch 2700:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.336297035217285 b: -17.12691307067871
    
    Epoch 2800:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.34121036529541 b: -17.15472412109375
    
    Epoch 2900:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.345354080200195 b: -17.17818260192871
    
    Epoch 3000:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.348850727081299 b: -17.197975158691406
    
    Epoch 3100:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.351799964904785 b: -17.214672088623047
    
    Epoch 3200:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.354288578033447 b: -17.228757858276367
    
    Epoch 3300:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.356387615203857 b: -17.24064064025879
    
    Epoch 3400:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.35815954208374 b: -17.250669479370117
    
    Epoch 3500:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.359652996063232 b: -17.259126663208008
    
    Epoch 3600:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.3609137535095215 b: -17.26626205444336
    
    Epoch 3700:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.3619771003723145 b: -17.272281646728516
    
    Epoch 3800:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.362875461578369 b: -17.277366638183594
    
    Epoch 3900:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.3636322021484375 b: -17.28165054321289
    
    Epoch 4000:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.3642706871032715 b: -17.28526496887207
    
    Epoch 4100:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.364809513092041 b: -17.288314819335938
    
    Epoch 4200:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.365264415740967 b: -17.290889739990234
    
    Epoch 4300:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.3656487464904785 b: -17.29306411743164
    
    Epoch 4400:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.365970134735107 b: -17.294885635375977
    
    Epoch 4500:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.366244792938232 b: -17.29644012451172
    
    Epoch 4600:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.36647367477417 b: -17.2977352142334
    
    Epoch 4700:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.366667747497559 b: -17.2988338470459
    
    Epoch 4800:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.36683464050293 b: -17.29977798461914
    
    Epoch 4900:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.366969108581543 b: -17.300540924072266
    
    Epoch 5000:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.367084503173828 b: -17.301193237304688
    


## Exploring PyTorch's available optimizers


```python
import torch.optim as optim
# torch's optim submodule has classes that implement optimization algorithms
```


```python
dir(optim)
```




    ['ASGD',
     'Adadelta',
     'Adagrad',
     'Adam',
     'AdamW',
     'Adamax',
     'LBFGS',
     'Optimizer',
     'RMSprop',
     'Rprop',
     'SGD',
     'SparseAdam',
     '__builtins__',
     '__cached__',
     '__doc__',
     '__file__',
     '__loader__',
     '__name__',
     '__package__',
     '__path__',
     '__spec__',
     'lr_scheduler']




```python
# example of vanilla gradient descent using the optim library

params = torch.tensor([1.0, 0.0], requires_grad=True)

opt = optim.SGD(params=[params], lr=1e-5)

opt
```




    SGD (
    Parameter Group 0
        dampening: 0
        lr: 1e-05
        momentum: 0
        nesterov: False
        weight_decay: 0
    )




```python
# SGD is ran over a subset of the input (a minibatch) in ideality, but unlike tensorflow, we can't just
# specify the batch_size in pytorch until now. Therefore, SGD will act like vanilla gradient descent instead of stochastic
# gradient descent when we feed it all the input.
```


```python
# testing the new optimizer

preds = model(x, *params)

curr_loss = loss(preds, truth_celsius)

curr_loss.backward() # obtaining the derivative of the loss with respect to the parameters

opt.step() # this is one of the 2 main functions of optimizers. It takes a step down the loss function
# since the optimizer was given the params variable in its deifnition, it updated the values of params
# when the step() function was called
# What happens is that the optimizer looks into params.grad and updates
# params, subtracting learning_rate times grad from it,

params
```




    tensor([ 9.5483e-01, -8.2600e-04], requires_grad=True)



# __WHEN WE PUT THE OPTIMIZER INTO THE LOOP, WE STILL NEED TO ZERO OUT THE GRADIENTS BEFORE THE BACKWARD() IS CALLED ON THE LOSS FUNCTION SO THAT THE GRADIENTS DO NOT ACCUMULUATE IN PARAMS.GRAD__


```python
def train_loop(epochs, opt, params, data, truth):

    for epoch in range(1, epochs+1):

        curr_pred = model(data, *params)
        
        curr_loss = loss(curr_pred, truth)
        
        opt.zero_grad() # zeroing the gradients before computing the derivates again
        # THIS IS THE 2ND MOST IMPORTANT FUNCTION OF AN OPTIMIZER
        # IT TAKES CARE OF ZEROING THE GRADIENTS OF THE PARAMETERS WITH WHICH THE OPTIMIZER WAS DEFINED
        
        curr_loss.backward() # computing the derivatives
        
        opt.step() # stepping down the loss, updating the parameters
        
        gradient = params.grad

        if epochs > 1000:
            if epoch % 100 == 0:
                print(f"Epoch {epoch}:\tloss = {torch.round(curr_loss)}\ngrad: dw: {torch.round(gradient[0])} db: {torch.round(gradient[1])}\nparams: w: {params[0]} b: {params[1]}\n")
        else:
            if epoch % 10 == 0:
                print(f"Epoch {epoch}:\tloss = {torch.round(curr_loss)}\ngrad: dw: {torch.round(gradient[0])} db: {torch.round(gradient[1])}\nparams: w: {params[0]} b: {params[1]}\n")

```


```python
params = torch.tensor([1.0, 0.0], requires_grad=True)
learning_rate = 1e-2
opt = torch.optim.SGD([params], lr=learning_rate)
    

train_loop(epochs=5000,
           opt=opt,
           params=params,
           data=x_n,
           truth=truth_celsius)
```

    Epoch 100:	loss = 22.0
    grad: dw: -0.0 db: 3.0
    params: w: 2.755274772644043 b: -2.5162105560302734
    
    Epoch 200:	loss = 17.0
    grad: dw: -0.0 db: 2.0
    params: w: 3.1637372970581055 b: -4.8284430503845215
    
    Epoch 300:	loss = 13.0
    grad: dw: -0.0 db: 2.0
    params: w: 3.508335590362549 b: -6.77915096282959
    
    Epoch 400:	loss = 10.0
    grad: dw: -0.0 db: 2.0
    params: w: 3.799055337905884 b: -8.424863815307617
    
    Epoch 500:	loss = 8.0
    grad: dw: -0.0 db: 1.0
    params: w: 4.044321060180664 b: -9.813268661499023
    
    Epoch 600:	loss = 6.0
    grad: dw: -0.0 db: 1.0
    params: w: 4.251237869262695 b: -10.984587669372559
    
    Epoch 700:	loss = 5.0
    grad: dw: -0.0 db: 1.0
    params: w: 4.425803184509277 b: -11.972770690917969
    
    Epoch 800:	loss = 5.0
    grad: dw: -0.0 db: 1.0
    params: w: 4.573074817657471 b: -12.806446075439453
    
    Epoch 900:	loss = 4.0
    grad: dw: -0.0 db: 1.0
    params: w: 4.697319507598877 b: -13.509775161743164
    
    Epoch 1000:	loss = 4.0
    grad: dw: -0.0 db: 1.0
    params: w: 4.802137851715088 b: -14.103133201599121
    
    Epoch 1100:	loss = 4.0
    grad: dw: -0.0 db: 0.0
    params: w: 4.890567779541016 b: -14.603717803955078
    
    Epoch 1200:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 4.9651713371276855 b: -15.02603530883789
    
    Epoch 1300:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.028110504150391 b: -15.382323265075684
    
    Epoch 1400:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.081210136413574 b: -15.682910919189453
    
    Epoch 1500:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.126006126403809 b: -15.936492919921875
    
    Epoch 1600:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.163797855377197 b: -16.15042495727539
    
    Epoch 1700:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.195681571960449 b: -16.33091163635254
    
    Epoch 1800:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.222580432891846 b: -16.48318099975586
    
    Epoch 1900:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.245273113250732 b: -16.61164093017578
    
    Epoch 2000:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.264420032501221 b: -16.720029830932617
    
    Epoch 2100:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.280571460723877 b: -16.811458587646484
    
    Epoch 2200:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.294197082519531 b: -16.88858985900879
    
    Epoch 2300:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.305691719055176 b: -16.95366096496582
    
    Epoch 2400:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.315390110015869 b: -17.008562088012695
    
    Epoch 2500:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.32357120513916 b: -17.054872512817383
    
    Epoch 2600:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.330473899841309 b: -17.093948364257812
    
    Epoch 2700:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.336297035217285 b: -17.12691307067871
    
    Epoch 2800:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.34121036529541 b: -17.15472412109375
    
    Epoch 2900:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.345354080200195 b: -17.17818260192871
    
    Epoch 3000:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.348850727081299 b: -17.197975158691406
    
    Epoch 3100:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.351799964904785 b: -17.214672088623047
    
    Epoch 3200:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.354288578033447 b: -17.228757858276367
    
    Epoch 3300:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.356387615203857 b: -17.24064064025879
    
    Epoch 3400:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.35815954208374 b: -17.250669479370117
    
    Epoch 3500:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.359652996063232 b: -17.259126663208008
    
    Epoch 3600:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.3609137535095215 b: -17.26626205444336
    
    Epoch 3700:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.3619771003723145 b: -17.272281646728516
    
    Epoch 3800:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.362875461578369 b: -17.277366638183594
    
    Epoch 3900:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.3636322021484375 b: -17.28165054321289
    
    Epoch 4000:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.3642706871032715 b: -17.28526496887207
    
    Epoch 4100:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.364809513092041 b: -17.288314819335938
    
    Epoch 4200:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.365264415740967 b: -17.290889739990234
    
    Epoch 4300:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.3656487464904785 b: -17.29306411743164
    
    Epoch 4400:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.365970134735107 b: -17.294885635375977
    
    Epoch 4500:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.366244792938232 b: -17.29644012451172
    
    Epoch 4600:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.36647367477417 b: -17.2977352142334
    
    Epoch 4700:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.366667747497559 b: -17.2988338470459
    
    Epoch 4800:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.36683464050293 b: -17.29977798461914
    
    Epoch 4900:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.366969108581543 b: -17.300540924072266
    
    Epoch 5000:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 5.367084503173828 b: -17.301193237304688
    


### Adam is a better optimizer than SGD. It is impervious to different scales in the parameters, sets the learning rate adaptively, and does not need the input to be normalized (will work with different scales between the input and the output).


```python
params = torch.tensor([1.0, 0.0], requires_grad=True)
learning_rate = 1e-1
opt = torch.optim.Adam([params], lr=learning_rate)
    

train_loop(epochs=2000,
           opt=opt,
           params=params,
           data=x, # USING THE DATA THAT IS NOT NORMALIZED
           truth=truth_celsius)
```

    Epoch 100:	loss = 24.0
    grad: dw: -22.0 db: 2.0
    params: w: 0.2608436942100525 b: -1.8626153469085693
    
    Epoch 200:	loss = 19.0
    grad: dw: -2.0 db: 2.0
    params: w: 0.2975502908229828 b: -3.744218349456787
    
    Epoch 300:	loss = 14.0
    grad: dw: -2.0 db: 2.0
    params: w: 0.33525633811950684 b: -5.882308483123779
    
    Epoch 400:	loss = 11.0
    grad: dw: -2.0 db: 2.0
    params: w: 0.373011976480484 b: -8.022296905517578
    
    Epoch 500:	loss = 8.0
    grad: dw: -1.0 db: 1.0
    params: w: 0.4080716371536255 b: -10.009542465209961
    
    Epoch 600:	loss = 6.0
    grad: dw: -1.0 db: 1.0
    params: w: 0.43885567784309387 b: -11.75447940826416
    
    Epoch 700:	loss = 4.0
    grad: dw: -1.0 db: 1.0
    params: w: 0.4646397531032562 b: -13.21601676940918
    
    Epoch 800:	loss = 4.0
    grad: dw: -0.0 db: 1.0
    params: w: 0.48533251881599426 b: -14.388969421386719
    
    Epoch 900:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 0.5012807846069336 b: -15.292988777160645
    
    Epoch 1000:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 0.5130982398986816 b: -15.962857246398926
    
    Epoch 1100:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 0.52152019739151 b: -16.44024658203125
    
    Epoch 1200:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 0.5272922515869141 b: -16.767436981201172
    
    Epoch 1300:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 0.5310946106910706 b: -16.982969284057617
    
    Epoch 1400:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 0.5335007309913635 b: -17.11935806274414
    
    Epoch 1500:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 0.5349613428115845 b: -17.2021541595459
    
    Epoch 1600:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 0.5358110070228577 b: -17.25031852722168
    
    Epoch 1700:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 0.5362839698791504 b: -17.277124404907227
    
    Epoch 1800:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 0.5365351438522339 b: -17.291364669799805
    
    Epoch 1900:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 0.5366626381874084 b: -17.298585891723633
    
    Epoch 2000:	loss = 3.0
    grad: dw: -0.0 db: 0.0
    params: w: 0.5367238521575928 b: -17.302061080932617
    


### Shuffling the dataset into training and validation sets


```python
m = x_n.size(0) # number of examples

val_split = int(0.2 * m)

perm = torch.randperm(m) # get shuffled indices for m elements


train_idx = perm[:-val_split] # take elements until val_split after which are val elements

validation_idx = perm[-val_split:]

train_idx, validation_idx
```




    (tensor([7, 5, 2, 8, 4, 0, 6, 3, 1]), tensor([ 9, 10]))




```python
train_x = x_n[train_idx] # normalizing with 0.1
train_y = truth_celsius[train_idx]

val_x = x_n[validation_idx] # normalizing
val_y = truth_celsius[validation_idx]
```


```python
def train_loop(epochs, opt, params, data, truth, valx, valy):
    
    for epoch in range(1, epochs+1):
        
        curr_loss = loss(model(data, *params), truth)
        # ======================================
        # validation loss
        val_loss = loss(model(valx, *params), valy)
        
        opt.zero_grad() # zeroing the gradients before computing the derivatives
        curr_loss.backward() # getting the derivates
        opt.step() # updating the parameters with the gradients

        if epoch % 500 == 0 or epoch <=3:
            print(f"Epoch {epoch}:\ttraining loss = {curr_loss} \
                    validation loss: {val_loss}\n \
                    grad: dw: {params.grad[0]} db: {params.grad[1]})\n \
                    params: w: {params[0]} b: {params[1]}\n")
        
```


```python
params = torch.tensor([1.0, 0.0], requires_grad=True)

learning_rate = 1e-2

optimizer = optim.SGD([params], lr=learning_rate)
# since we use SGD here we needed the normalized inputs before

train_loop(
    epochs = 3000,
    opt = optimizer,
    params = params,
    data = train_x,
    truth = train_y,
    valx = val_x,
    valy = val_y)
```

    Epoch 1:	training loss = 70.56228637695312                     validation loss: 124.47359466552734
                         grad: dw: -63.99653625488281 db: -8.311111450195312)
                         params: w: 1.639965295791626 b: 0.08311111479997635
    
    Epoch 2:	training loss = 40.37565612792969                     validation loss: 51.57497024536133
                         grad: dw: -29.019189834594727 db: -1.8732287883758545)
                         params: w: 1.93015718460083 b: 0.10184340178966522
    
    Epoch 3:	training loss = 34.22074890136719                     validation loss: 30.382097244262695
                         grad: dw: -13.344480514526367 db: 1.0081175565719604)
                         params: w: 2.0636019706726074 b: 0.0917622298002243
    
    Epoch 500:	training loss = 7.120349407196045                     validation loss: 6.405239582061768
                         grad: dw: -0.23494958877563477 db: 1.2752008438110352)
                         params: w: 4.133184909820557 b: -10.560690879821777
    
    Epoch 1000:	training loss = 3.412811040878296                     validation loss: 4.223626136779785
                         grad: dw: -0.08882725238800049 db: 0.48215439915657043)
                         params: w: 4.883641242980957 b: -14.633679389953613
    
    Epoch 1500:	training loss = 2.882786750793457                     validation loss: 3.7479934692382812
                         grad: dw: -0.033600687980651855 db: 0.18230119347572327)
                         params: w: 5.16738748550415 b: -16.173669815063477
    
    Epoch 2000:	training loss = 2.807011842727661                     validation loss: 3.618082284927368
                         grad: dw: -0.01270151138305664 db: 0.06892913579940796)
                         params: w: 5.274672508239746 b: -16.755943298339844
    
    Epoch 2500:	training loss = 2.796180009841919                     validation loss: 3.576099395751953
                         grad: dw: -0.00479435920715332 db: 0.026062488555908203)
                         params: w: 5.315238952636719 b: -16.976110458374023
    
    Epoch 3000:	training loss = 2.7946300506591797                     validation loss: 3.561248779296875
                         grad: dw: -0.0018211603164672852 db: 0.009854048490524292)
                         params: w: 5.330574035644531 b: -17.059341430664062
    


### For larger neural networks, having to make a computation graph for the validation set is useless, becase we never use that loss to compute the derivatives. So we could (__and should__) forgo the autograd use for the validation loss as follows.

# __TORCH.NO_GRAD()__


```python
with torch.no_grad(): # THIS WAY PYTORCH SWITCHES AUTOGRAD OFF FOR ALL EXECUTIONS IN THIS SECTION
    val_preds = model(val_t_u, *params)
    val_loss = loss_fn(val_preds, val_truth)
    assert val_loss.requires_grad == False # Checks that our output requires_grad args are forced to False inside this block
```

### We could also set the grad to False when we just need to compute the forward pass without backpropagation, which would not require the gradient.

# __TORCH.SET_GRAD_ENABLED(TRUE / FALSE)__


```python
def calc_forward(data, truth, is_train):
    with torch.set_grad_enabled(is_train): # is_train is a boolean
    preds = model(data, *params)
    loss = loss_fn(preds, truth)
    return loss
```

## Exercise

1. Redefine the model to be $w_1 x + w_2 x^2 + b$.
  
  1. What parts of the training loop, and so on, need to change to accommodate
this redefinition?
        1. Since only the model / hypothesis function changes, the way the parameteres are initialized will need to be modified. Apart from that, since the backpropagation is automated with autograd, no other changes would be required. The loss function would remain the same since the hypothesis is still linear regression.
  2. What parts are agnostic to swapping out the model?
        1. Uhhh.
  3. Is the resulting loss higher or lower after training?
        1. Resulting loss is magnitudes higher than the previous model definition. Even a lower learning rate had to be used to prevent the loss from overflowing.
  4. Is the actual result better or worse?
        1. Worse. The new quadratic model seems to be having a hard time fitting to a linear dataset.


```python
def model(data, w1, w2, b):
    return w1 * data + w2 * torch.pow(data, 2) + b
```


```python
params = torch.tensor([1.0, 1.0, 0.0], requires_grad=True)

learning_rate = 1e-4

optimizer = optim.SGD([params], lr=learning_rate)
# since we use SGD here we needed the normalized inputs before

train_loop(
    epochs = 5000,
    opt = optimizer,
    params = params,
    data = train_x,
    truth = train_y,
    valx = val_x,
    valy = val_y)
```

    Epoch 1:	training loss = 610.8646850585938                     validation loss: 967.9776000976562
                         grad: dw: 252.61439514160156 db: 1547.559814453125)
                         params: w: 0.9747385382652283 b: 0.8452440500259399
    
    Epoch 2:	training loss = 390.2024230957031                     validation loss: 599.0106201171875
                         grad: dw: 202.22427368164062 db: 1227.56298828125)
                         params: w: 0.9545161128044128 b: 0.72248774766922
    
    Epoch 3:	training loss = 251.28829956054688                     validation loss: 369.36883544921875
                         grad: dw: 162.24293518066406 db: 973.6708374023438)
                         params: w: 0.9382918477058411 b: 0.6251206398010254
    
    Epoch 500:	training loss = 11.47903823852539                     validation loss: 5.464059352874756
                         grad: dw: 7.0791335105896 db: -1.199753761291504)
                         params: w: 0.48869889974594116 b: 0.31673797965049744
    
    Epoch 1000:	training loss = 8.941032409667969                     validation loss: 4.740511417388916
                         grad: dw: 5.750217437744141 db: -0.9763708114624023)
                         params: w: 0.16916008293628693 b: 0.3709403872489929
    
    Epoch 1500:	training loss = 7.25091028213501                     validation loss: 4.18851375579834
                         grad: dw: 4.666435718536377 db: -0.7942590713500977)
                         params: w: -0.09028209000825882 b: 0.4150415360927582
    
    Epoch 2000:	training loss = 6.124751567840576                     validation loss: 3.7636570930480957
                         grad: dw: 3.7826056480407715 db: -0.6456022262573242)
                         params: w: -0.30071356892585754 b: 0.4509047269821167
    
    Epoch 2500:	training loss = 5.373710632324219                     validation loss: 3.433997631072998
                         grad: dw: 3.06180739402771 db: -0.5244884490966797)
                         params: w: -0.4711754620075226 b: 0.4800494611263275
    
    Epoch 3000:	training loss = 4.8721842765808105                     validation loss: 3.176340341567993
                         grad: dw: 2.4740002155303955 db: -0.4256629943847656)
                         params: w: -0.6090417504310608 b: 0.5037150382995605
    
    Epoch 3500:	training loss = 4.536623001098633                     validation loss: 2.9736461639404297
                         grad: dw: 1.9946339130401611 db: -0.34513092041015625)
                         params: w: -0.7203262448310852 b: 0.5229122042655945
    
    Epoch 4000:	training loss = 4.311458587646484                     validation loss: 2.8132803440093994
                         grad: dw: 1.6037170886993408 db: -0.27947235107421875)
                         params: w: -0.8099337220191956 b: 0.5384653210639954
    
    Epoch 4500:	training loss = 4.159730911254883                     validation loss: 2.6857800483703613
                         grad: dw: 1.2849442958831787 db: -0.2258930206298828)
                         params: w: -0.8818632960319519 b: 0.5510465502738953
    
    Epoch 5000:	training loss = 4.056850433349609                     validation loss: 2.58396577835083
                         grad: dw: 1.0250134468078613 db: -0.18211936950683594)
                         params: w: -0.9393784999847412 b: 0.5612044334411621
    


c6302967ee0a8fd78197b50c63deee20858d74b7
ipynb
Jupyter Notebook
Chapter_5/Chapter_5.ipynb
ramanshsharma2806/pytorch_practice
1a6b331d26d1cc37a8882fc12b1fa63d4d3ba8e9
MIT
2020-09-16T14:38:28.000Z
2020-09-16T14:38:28.000Z
Chapter_5/Chapter_5.ipynb
ramanshsharma2806/pytorch_practice
1a6b331d26d1cc37a8882fc12b1fa63d4d3ba8e9
MIT
Chapter_5/Chapter_5.ipynb
ramanshsharma2806/pytorch_practice
1a6b331d26d1cc37a8882fc12b1fa63d4d3ba8e9
MIT
Qwen/Qwen-72B
1. YES
2. YES