# Introduction to Deep Learning with PyTorch

In this notebook, you'll get introduced to [PyTorch](http://pytorch.org/), a framework for building and training neural networks. PyTorch in a lot of ways behaves like the arrays you love from Numpy. These Numpy arrays, after all, are just tensors. PyTorch takes these tensors and makes it simple to move them to GPUs for the faster processing needed when training neural networks. It also provides a module that automatically calculates gradients (for backpropagation!) and another module specifically for building neural networks. All together, PyTorch ends up being more coherent with Python and the Numpy/Scipy stack compared to TensorFlow and other frameworks.



## Neural Networks

Deep Learning is based on artificial neural networks which have been around in some form since the late 1950s. The networks are built from individual parts approximating neurons, typically called units or simply "neurons." Each unit has some number of weighted inputs. These weighted inputs are summed together (a linear combination) then passed through an activation function to get the unit's output.



Mathematically this looks like: 

$$
\begin{align}
y &= f(w_1 x_1 + w_2 x_2 + b) \\
y &= f\left(\sum_i w_i x_i \right)
\end{align}
$$

With vectors this is the dot/inner product of two vectors:

$$
h = \begin{bmatrix}
x_1 \, x_2 \cdots  x_n
\end{bmatrix}
\cdot 
\begin{bmatrix}
           w_1 \\
           w_2 \\
           \vdots \\
           w_n
\end{bmatrix}
$$

### Stack them up!

We can assemble these unit neurons into layers and stacks, into a network of neurons. The output of one layer of neurons becomes the input for the next layer. With multiple input units and output units, we now need to express the weights as a matrix.



We can express this mathematically with matrices again and use matrix multiplication to get linear combinations for each unit in one operation. For example, the hidden layer ($h_1$ and $h_2$ here) can be calculated 

$$
\vec{h} = [h_1 \, h_2] = 
\begin{bmatrix}
x_1 \, x_2 \cdots \, x_n
\end{bmatrix}
\cdot 
\begin{bmatrix}
           w_{11} & w_{12} \\
           w_{21} &w_{22} \\
           \vdots &\vdots \\
           w_{n1} &w_{n2}
\end{bmatrix}
$$

The output for this small network is found by treating the hidden layer as inputs for the output unit. The network output is expressed simply

$$
y =  f_2 \! \left(\, f_1 \! \left(\vec{x} \, \mathbf{W_1}\right) \mathbf{W_2} \right)
$$

## Tensors

It turns out neural network computations are just a bunch of linear algebra operations on *tensors*, a generalization of matrices. A vector is a 1-dimensional tensor, a matrix is a 2-dimensional tensor, an array with three indices is a 3-dimensional tensor (RGB color images for example). The fundamental data structure for neural networks are tensors and PyTorch (as well as pretty much every other deep learning framework) is built around tensors.



With the basics covered, it's time to explore how we can use PyTorch to build a simple neural network.


```python
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import numpy as np
import torch
```

First, let's see how we work with PyTorch tensors. These are the fundamental data structures of neural networks and PyTorch, so it's imporatant to understand how these work.


```python
x = torch.rand(3, 2)
x.size()
```




    torch.Size([3, 2])




```python
y = torch.ones(x.size())
y
```




    tensor([[1., 1.],
            [1., 1.],
            [1., 1.]])




```python
z = x + y
z
```




    tensor([[1.0227, 1.0487],
            [1.4433, 1.3408],
            [1.1525, 1.4355]])



In general PyTorch tensors behave similar to Numpy arrays. They are zero indexed and support slicing.


```python
z[0]
```




    tensor([1.0227, 1.0487])




```python
z[:, 1:]
```




    tensor([[1.0487],
            [1.3408],
            [1.4355]])



Tensors typically have two forms of methods, one method that returns another tensor and another method that performs the operation in place. That is, the values in memory for that tensor are changed without creating a new tensor. In-place functions are always followed by an underscore, for example `z.add()` and `z.add_()`.


```python
# Return a new tensor z + 1
z.add(1)
```




    tensor([[2.0227, 2.0487],
            [2.4433, 2.3408],
            [2.1525, 2.4355]])




```python
# z tensor is unchanged
z
```




    tensor([[1.0227, 1.0487],
            [1.4433, 1.3408],
            [1.1525, 1.4355]])




```python
# Add 1 and update z tensor in-place
z.add_(1)
```




    tensor([[2.0227, 2.0487],
            [2.4433, 2.3408],
            [2.1525, 2.4355]])




```python
# z has been updated
z
```




    tensor([[2.0227, 2.0487],
            [2.4433, 2.3408],
            [2.1525, 2.4355]])



### Reshaping

Reshaping tensors is a really common operation. First to get the size and shape of a tensor use `.size()`. Then, to reshape a tensor, use `.resize_()`. Notice the underscore, reshaping is an in-place operation.


```python
z.size()
```




    torch.Size([3, 2])




```python
z.resize_(2, 3)
```




    tensor([[2.0227, 2.0487, 2.4433],
            [2.3408, 2.1525, 2.4355]])




```python
z
```




    tensor([[2.0227, 2.0487, 2.4433],
            [2.3408, 2.1525, 2.4355]])



## Numpy to Torch and back

Converting between Numpy arrays and Torch tensors is super simple and useful. To create a tensor from a Numpy array, use `torch.from_numpy()`. To convert a tensor to a Numpy array, use the `.numpy()` method.


```python
a = np.random.rand(4,3)
a
```




    array([[0.71561694, 0.49365764, 0.42654007],
           [0.38610867, 0.73036579, 0.40426613],
           [0.30673351, 0.72503498, 0.343875  ],
           [0.48630183, 0.74443254, 0.40740172]])




```python
b = torch.from_numpy(a)
b
```




    tensor([[0.7156, 0.4937, 0.4265],
            [0.3861, 0.7304, 0.4043],
            [0.3067, 0.7250, 0.3439],
            [0.4863, 0.7444, 0.4074]], dtype=torch.float64)




```python
b.numpy()
```




    array([[0.71561694, 0.49365764, 0.42654007],
           [0.38610867, 0.73036579, 0.40426613],
           [0.30673351, 0.72503498, 0.343875  ],
           [0.48630183, 0.74443254, 0.40740172]])



The memory is shared between the Numpy array and Torch tensor, so if you change the values in-place of one object, the other will change as well.


```python
# Multiply PyTorch Tensor by 2, in place
b.mul_(2)
```




    tensor([[1.4312, 0.9873, 0.8531],
            [0.7722, 1.4607, 0.8085],
            [0.6135, 1.4501, 0.6877],
            [0.9726, 1.4889, 0.8148]], dtype=torch.float64)




```python
# Numpy array matches new values from Tensor
a
```




    array([[1.43123389, 0.98731528, 0.85308015],
           [0.77221734, 1.46073158, 0.80853226],
           [0.61346702, 1.45006997, 0.68775   ],
           [0.97260366, 1.48886509, 0.81480343]])



## Feedforward neural networks with PyTorch

Next I'll show you how to use this pattern to build a neural network with PyTorch. First up, we need to get our dataset. This is provided through the `torchvision` package. The code below will download the MNIST dataset, then create training and test datasets for us.


```python
import matplotlib.pyplot as plt
from torchvision import datasets, transforms
```


```python
# Define a transform to normalize the data
transform = transforms.Compose([transforms.ToTensor(),
                              transforms.Normalize((0.5,), (0.5,)),
                             ])
# Download and load the training data
trainset = datasets.MNIST('MNIST_data/', download=True, train=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)

# Download and load the test data
testset = datasets.MNIST('MNIST_data/', download=True, train=False, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=64, shuffle=True)
```

    Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
    Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to MNIST_data/MNIST/raw/train-images-idx3-ubyte.gz



      0%|          | 0/9912422 [00:00<?, ?it/s]


    Extracting MNIST_data/MNIST/raw/train-images-idx3-ubyte.gz to MNIST_data/MNIST/raw
    
    Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
    Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to MNIST_data/MNIST/raw/train-labels-idx1-ubyte.gz



      0%|          | 0/28881 [00:00<?, ?it/s]


    Extracting MNIST_data/MNIST/raw/train-labels-idx1-ubyte.gz to MNIST_data/MNIST/raw
    
    Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
    Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to MNIST_data/MNIST/raw/t10k-images-idx3-ubyte.gz



      0%|          | 0/1648877 [00:00<?, ?it/s]


    Extracting MNIST_data/MNIST/raw/t10k-images-idx3-ubyte.gz to MNIST_data/MNIST/raw
    
    Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
    Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to MNIST_data/MNIST/raw/t10k-labels-idx1-ubyte.gz



      0%|          | 0/4542 [00:00<?, ?it/s]


    Extracting MNIST_data/MNIST/raw/t10k-labels-idx1-ubyte.gz to MNIST_data/MNIST/raw
    



```python
dataiter = iter(trainloader)
images, labels = dataiter.next()
```

We have the training data loaded into `trainloader` and we make that an iterator with `iter(trainloader)`. We'd use this to loop through the dataset for training, but here I'm just grabbing the first batch so we can check out the data. We can see below that `images` is just a tensor with size (64, 1, 28, 28). So, 64 images per batch, 1 color channel, and 28x28 images.


```python
images.size()
```




    torch.Size([64, 1, 28, 28])




```python
plt.imshow(images[1].numpy().squeeze(), cmap='Greys_r')
```

## Building networks with PyTorch

Here I'll use PyTorch to build a simple feedfoward network to classify the MNIST images. That is, the network will receive a digit image as input and predict the digit in the image.



To build a neural network with PyTorch, you use the `torch.nn` module. The network itself is a class inheriting from `torch.nn.Module`. You define each of the operations separately, like `nn.Linear(784, 128)` for a fully connected linear layer with 784 inputs and 128 units.

The class needs to include a `forward` method that implements the forward pass through the network. In this method, you pass some input tensor `x` through each of the operations you defined earlier. The `torch.nn` module also has functional equivalents for things like ReLUs in `torch.nn.functional`. This module is usually imported as `F`. Then to use a ReLU activation on some layer (which is just a tensor), you'd do `F.relu(x)`. Below are a few different commonly used activation functions.



So, for this network, I'll build it with three fully connected layers, then a softmax output for predicting classes. The softmax function is similar to the sigmoid in that it squashes inputs between 0 and 1, but it's also normalized so that all the values sum to one like a proper probability distribution.


```python
from torch import nn
from torch import optim
import torch.nn.functional as F
```


```python
class Network(nn.Module):
    def __init__(self):
        super().__init__()
        # Defining the layers, 128, 64, 10 units each
        self.fc1 = nn.Linear(784, 128)
        self.fc2 = nn.Linear(128, 64)
        # Output layer, 10 units - one for each digit
        self.fc3 = nn.Linear(64, 10)
        
    def forward(self, x):
        ''' Forward pass through the network, returns the output logits '''
        
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        x = F.relu(x)
        x = self.fc3(x)
        
        return x
    
    def predict(self, x):
        ''' This function for predicts classes by calculating the softmax '''
        logits = self.forward(x)
        return F.softmax(logits)

net = Network()
net
```




    Network(
      (fc1): Linear(in_features=784, out_features=128, bias=True)
      (fc2): Linear(in_features=128, out_features=64, bias=True)
      (fc3): Linear(in_features=64, out_features=10, bias=True)
    )



### Initializing weights and biases

The weights and such are automatically initialized for you, but it's possible to customize how they are initialized. The weights and biases are tensors attached to the layer you defined, you can get them with `net.fc1.weight` for instance.


```python
print(net.fc1.weight)
print(net.fc1.bias)
```

    Parameter containing:
    tensor([[-0.0057, -0.0222, -0.0297,  ...,  0.0106,  0.0173, -0.0225],
            [-0.0339,  0.0224, -0.0163,  ..., -0.0079,  0.0323,  0.0258],
            [ 0.0109, -0.0319, -0.0306,  ..., -0.0210, -0.0067,  0.0306],
            ...,
            [-0.0046, -0.0081,  0.0092,  ...,  0.0042, -0.0266,  0.0322],
            [-0.0012,  0.0152, -0.0134,  ...,  0.0292, -0.0341,  0.0243],
            [ 0.0046, -0.0309, -0.0150,  ..., -0.0297, -0.0310,  0.0136]],
           requires_grad=True)
    Parameter containing:
    tensor([-0.0004, -0.0107,  0.0340,  0.0038, -0.0255,  0.0248, -0.0322,  0.0329,
             0.0173, -0.0025,  0.0151,  0.0064,  0.0170, -0.0010,  0.0085,  0.0134,
             0.0218,  0.0333, -0.0140,  0.0053, -0.0346,  0.0312,  0.0322,  0.0046,
             0.0344, -0.0294, -0.0068, -0.0018, -0.0324,  0.0271,  0.0263,  0.0123,
            -0.0278,  0.0357, -0.0165,  0.0255,  0.0143,  0.0134, -0.0034, -0.0048,
             0.0292, -0.0210,  0.0116,  0.0053,  0.0032,  0.0317,  0.0139, -0.0245,
            -0.0308,  0.0148,  0.0132, -0.0287, -0.0030,  0.0274, -0.0146, -0.0112,
             0.0143, -0.0158,  0.0348, -0.0047, -0.0076,  0.0105,  0.0290,  0.0119,
             0.0177, -0.0177,  0.0037,  0.0258,  0.0187, -0.0307, -0.0141, -0.0047,
            -0.0284,  0.0166,  0.0218,  0.0156,  0.0018,  0.0190, -0.0153,  0.0005,
            -0.0234,  0.0141,  0.0308, -0.0212, -0.0332,  0.0209, -0.0095, -0.0133,
            -0.0253, -0.0102,  0.0297,  0.0129,  0.0206, -0.0304,  0.0304,  0.0333,
            -0.0035,  0.0146, -0.0260,  0.0345,  0.0121, -0.0099, -0.0237,  0.0070,
             0.0276, -0.0247, -0.0053,  0.0221, -0.0091, -0.0124, -0.0283, -0.0100,
             0.0251, -0.0169, -0.0142, -0.0002,  0.0275, -0.0118,  0.0345,  0.0324,
             0.0232,  0.0227, -0.0022, -0.0108, -0.0273, -0.0060,  0.0256,  0.0102],
           requires_grad=True)


For custom initialization, we want to modify these tensors in place. Once we have the tensors, we can fill them with zeros (for biases) or random normal values.


```python
# Set biases to all zeros
net.fc1.bias.data.fill_(0)
```




    tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
            0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
            0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
            0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
            0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
            0., 0., 0., 0., 0., 0., 0., 0.])




```python
# sample from random normal with standard dev = 0.01
net.fc1.weight.data.normal_(std=0.01)
```




    tensor([[ 6.4400e-03,  1.0824e-02,  1.0809e-02,  ..., -1.0652e-03,
              4.1703e-03,  2.0442e-02],
            [-2.8747e-02, -7.4504e-03,  1.6087e-03,  ...,  1.6905e-04,
             -1.6302e-03, -5.7481e-03],
            [-1.6169e-03,  8.1673e-03,  1.6747e-02,  ..., -1.0482e-02,
              1.9876e-03,  8.7329e-03],
            ...,
            [-7.5507e-03, -9.2924e-03,  6.6852e-03,  ..., -1.0514e-02,
              1.7884e-02,  1.3018e-02],
            [ 9.7859e-03, -1.6901e-03,  5.5006e-05,  ...,  4.0575e-03,
             -1.3077e-02,  4.5288e-06],
            [-2.5877e-02, -5.0783e-03,  1.5483e-03,  ...,  7.1621e-03,
              5.5963e-04,  4.8401e-03]])



### Forward pass

Now that we have a network, let's see what happens when we pass in an image.

For a better visualization, we define the following function


```python
def view_classify(img, ps):
    ''' Function for viewing an image and it's predicted classes.
    '''
    ps = ps.data.numpy().squeeze()

    fig, (ax1, ax2) = plt.subplots(figsize=(6,7), ncols=2)
    ax1.imshow(img.numpy().squeeze())
    ax1.axis('off')
    ax2.barh(np.arange(10), ps)
    ax2.set_aspect(0.1)
    ax2.set_yticks(np.arange(10))
    ax2.set_yticklabels(np.arange(10).astype(int), size='large');
    ax2.set_title('Digit Probability')
    ax2.set_xlim(0, 1.1)
```


```python
# Grab some data 
dataiter = iter(trainloader)
inputs, labels = dataiter.next()
print(inputs.size(), labels.size())
inputs.resize_(64, 1, 784)

# Forward pass through the network
img_idx = 0
logits = net(inputs[img_idx,:])

# Predict the class from the network output
ps = F.softmax(logits)

img = images[img_idx]
view_classify(img.resize_(1, 28, 28), ps)
```


```python
inputs.size()
```




    torch.Size([64, 1, 784])



As you can see above, our network has basically no idea what this digit is. It's because we haven't trained it yet, all the weights are random! Next up, we'll look at training this network so it learns how to properly classify these digits.

dc5e6f321adffb193df8c8c1fea4c671cb0770ce
ipynb
Jupyter Notebook
homeworks/hw2/Tutorial 1.ipynb
baixianger/DLAV-2022
4c43caa0da1b48c5c7320c0923bac0b33a9e9b04
MIT
homeworks/hw2/Tutorial 1.ipynb
baixianger/DLAV-2022
4c43caa0da1b48c5c7320c0923bac0b33a9e9b04
MIT
homeworks/hw2/Tutorial 1.ipynb
baixianger/DLAV-2022
4c43caa0da1b48c5c7320c0923bac0b33a9e9b04
MIT
Qwen/Qwen-72B
1. YES
2. YES