We start by bringing in the Python libraries that we will be using for this code. We will be using Torch, which will enable us to run the code in the GPU for faster processing as compared to the CPU. Torch also assists us during backpropagation, using Autograd that does differentiation and stores the gradients that can be called when required. PyTorch also enables us to create neural networks very easily, and PyTorch together with the Numpy, Pandas and Scipy libraries makes it very powerful for Neural Networks calculations.

## Neural Networks

Neural networks are the building blocks for deep learning. They are made up of neurons or units. Each weighted unit sums up the units of the previous layers and passes the sum through an activation function to get the neuron's output. Some activation functions used in this code are sigmoid, ReLU (Rectified Linear Units), Softmax, tanH (hyperbolic tangent) just to name a few. 

 

A simple neural network would be depicted as shown below: 

$$
\begin{align}
y &= f(w_1 x_1 + w_2 x_2 + w_3 x_3 + b) \\
y &= f\left(\sum_i w_i x_i +b \right)
\end{align}
$$

The linear transformation done on each unit is given by the dot product of the input vector and the weight matrix:

$$
h = \begin{bmatrix}
x_1 \, x_2 \cdots  x_n
\end{bmatrix}
\cdot 
\begin{bmatrix}
           w_{11}  w_{21}\\
           w_{12}  w_{22}\\
           \vdots \\
           w_{1n}  w_{2n}
\end{bmatrix}
$$

We begin by importing the libraries that we will be using in our code:
* torch - used to bring the torch package which will be used in creating the neural network layers, weihts and biases, as well as the backpropagation using autograd.
* nn - this module will be used for easy neural network creation.
* optim - used to optimize the neural network to adjust the weights to the optimal weights to give us the intended labels.



```python
import torch 
from torch import nn, optim
from torch import optim
import numpy as np
```

We're going to build a small neural network with only one example as shown below. Our goal is to build a network that will give the correct output `[0.0,1.0,0.0]`


```python
x = torch.tensor([0.1,0.2,0.7])
x = x.reshape(1,3)
print(x.shape)
print(x)
```

    torch.Size([1, 3])
    tensor([[0.1000, 0.2000, 0.7000]])


Our output will be given by the vector below:


```python
labels = torch.tensor([0,1,0])
labels = labels.reshape(1,3)
labels = labels
print(labels.shape)
print(labels)

```

    torch.Size([1, 3])
    tensor([[0, 1, 0]])


Here is where the neural network is made and all the fun begins! I will go through the code line by line.

```python
class Network(nn.Module):
```

Using the `nn.Module` combined with `__super().__init__()` creates a class that will assist us with tracking of the neural network methods and attributes. The class name can be changed to anything, although it is mandatory to inherit it from the `nn.Module`.

```python
self.hidden = nn.Linear(3,3)
```

This line creates a 3 X 3 matrix for the first hidden layer. `nn.Linear(3,3)` creates a linear transformation $x\mathbf{W} + b$, with 3 inputs and 3 outputs and assigns it to `self.hidden`. This module also creates the weights and biases which will be used in the feedforward using the `forward` method. 

The subsequent line, `self.hidden2 = nn.Linear(3,3)` performs the same function as the first hidden layer. Here we generate a 3 X 3 tensor for the second hidden layer. A linear transformation creates the weights and bias tensors and assigns them to `self.hidden2`. This will later be used in the `forward` method.

```python
self.output = nn.Linear(3,3)
```
This creates a linear transformation in the output of size 3 X 3 to be used to generate the output of the neural network. This will be used in the softmax activation function in the output layer.

```python
self.hidden.weight = torch.nn.Parameter(torch.tensor([[0.1,0.3,0.4],[0.2,0.2,0.3],[0.3,0.7,0.9]]))
```

This line of code creates the exact weights that will be used in the code. This was done to enable comparison between the code and the hand calculations already performed.


`self.hidden2.weight and self.output.weight` tensors are also created to explicitly define the tensors for the second hidden layer and the output layer of the neural network.

```python
self.sigmoid = nn.Sigmoid()
self.relu = nn.ReLU()
self.softmax = nn.Softmax(dim=1)
```
These operations are the sigmoid, Rectified Linear Unit (ReLU) and the softmax activation functions. The softmax is used in the output where we will be generating 3 classes for the 3 output units. The `dim=1` is used to ensure that the summation is done across the columns, as opposed to across the rows. The softmax summation should always add to 1 as this is a propability summation.

Some things to note :
* The softmax activation function can ONLY be used in the output layer. 
* The softmax function and the Logistic/sigmoid function yield the same results when the number of classes = 2, therefore softmax is a generalization of the sigmoid function.

```python
def forward (self,x):
```

PyTorch neural networks require a `forward` method defined to be able to perform the feedforward step. This is achieved by taking a tensor `x` and taking it through all the operation defined in the `__init__` method.

```python
x = self.hidden(x)
    
x = self.relu(x)
       
x = self.hidden2(x)
        
x = self.sigmoid(x)
        
x = self.output(x)
        
x = self.softmax(x)
```

Here we pass the input tensor `x` through the hidden layes, hidden layer 2, the sigmoid the output layer and finally the softmax function. The operations must be sequenced correctly in the forward method to ensure that the operations are done sequentially as expected.



```python
class Network(nn.Module):
    def __init__(self):
        super().__init__()
        
        # Inputs to hidden layer linear transformation
        self.hidden = nn.Linear(3, 3)
        self.hidden2 = nn.Linear(3, 3)
        self.output = nn.Linear(3, 3)
        
        self.hidden.weight = torch.nn.Parameter(torch.tensor([[0.1,0.3,0.4],[0.2,0.2,0.3],[0.3,0.7,0.9]]))
        self.hidden2.weight = torch.nn.Parameter(torch.tensor([[0.2,0.3,0.6],[0.3,0.5,0.4],[0.5,0.7,0.8]]))
        self.output.weight = torch.nn.Parameter(torch.tensor([[0.1,0.3,0.5],[0.4,0.7,0.2],[0.8,0.2,0.9]]))
       
        
        
        # Define sigmoid activation, ReLU and softmax output 
        self.sigmoid = nn.Sigmoid()
        self.relu = nn.ReLU()
        self.softmax = nn.Softmax(dim=1)
        
    def forward(self, x):
        # Pass the input tensor through each of our operations
        x = self.hidden(x)
    
        x = self.relu(x)
       
        x = self.hidden2(x)
        
        x = self.sigmoid(x)
        
        x = self.output(x)
        
        x = self.softmax(x)
        
        return x

    
    
```

We can now create a `Network` object `model = Network()`

An epoch in Machine Learning is the number of times we want to pass through the same data to optimize the weights to reduce the loss. For this neural network, we will use 50 epochs. 

The next thing after the feedforward is the calculation of the loss of the neural network. We will be using the `nn.CrosEntropyloss()` criterion. This criterion is a combination of the `nn.LogSoftmax()` and `nn.NLLLoss`. The NLLLoss stands for the Negative Log Likelihood Loss. It is used when softmax function is being used in the output layer of the neural network. This loss maximizes the likelihood of the correct label while reducing the other classes to 0.

`loss.backward` performs the backpropagation and calculates the gradients in each weight and bias.


PyTorch has a module, autograd that enables us to automatically calculate the gradients of the tensors. We can use this to calculate the gradients with all of our parameters to the loss. Autograd tracks the operations performed during the feedforward, then calculates the gradient dugin backpropagation. Autograd, basically does differentiation, for those who are familiar with math Calculus using chain rule. During backpropagation, it is important to ensure that the gradient calculations are turned on by setting `requires_grad = True` on a tensor. This can be done during the tensor creation with `required_grad` keyword, or any time using `y.required_grad_True)`.

Gradient can be trurned off in a block of code using `torch.no_grad()` or globally on the entire code using `torch.set_grad_enables(True|False)`.

To train the network, we use the [`optim` package](https://pytorch.org/docs/stable/optim.html). For this network, we use the `optim.Adam` package. There are a lot of other packages we can use such as `optim.SGD` where the opimizer used Stochastic gradient descent for optimization. It is critical to zero out the gradient in each epoch using `optimizer.zero_grad()` to ensure that the gradients are not accumulated for each training .For this network, the learning rate, `lr=0.09` was adequate enough to ensure that we got to a global minumum as fast as possible.



```python
model = Network()
epochs = 50
for t in range (1,epochs):
    criterion = nn.CrossEntropyLoss()
    logps = model(x)
    print(labels)
    loss = criterion(logps, torch.max(labels, 1)[1])
    loss.backward()
    optimizer = optim.Adam(model.parameters(), lr=0.09)
    optimizer.zero_grad()
    optimizer.step()
    print(loss)
    print(model.forward(x))
```

    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)
    tensor([[0, 1, 0]])
    tensor(1.2714, grad_fn=<NllLossBackward>)
    tensor([[0.2725, 0.1738, 0.5537]], grad_fn=<SoftmaxBackward>)


Below I take you through a quick example showing how autograd calculates and performs backpropagation. 


```python
a = torch.randn(2,2, requires_grad=True)
print(a)
```

    tensor([[-0.3680, -1.1311],
            [ 0.1723, -1.1959]], requires_grad=True)



```python
y = a**3
print(y)
```

    tensor([[-0.0498, -1.4470],
            [ 0.0051, -1.7103]], grad_fn=<PowBackward0>)


Here, we see that `y` is created with the a power operation `PowBackward0`


```python
m = y**2
print(m)
```

    tensor([[2.4846e-03, 2.0937e+00],
            [2.6132e-05, 2.9250e+00]], grad_fn=<PowBackward0>)


The `y` then goes through another power operation where it is squared.


```python
print(m.grad_fn)
```

    <PowBackward0 object at 0x12027d990>


Autograd tracks all the operations and calculates the gradients for each operation during backpropagation. Next we take a mean operation to get a scalar value from m.


```python
z = m.mean()
print(z)
```

    tensor(0.1634, grad_fn=<MeanBackward0>)



```python
print(a.grad)
#print((a**2)/3)
```

    None


We can see that there are no gradients since no backpropagation has been done at this point.The gradient after the `z.backward()` results in exactly the same results from the math differentiation with respect to a:
$$
\frac{\partial z}{\partial a} = \frac{\partial}{\partial a}\left[\frac{1}{n}\sum_i^n (a_i^3)^2\right] = \frac{3a^5}{2}
$$


```python
z.backward()
print(a.grad)
print((3*(a**5))/2)
```

    tensor([[ 1.6507e-04,  4.4403e-01],
            [-3.4622e-03,  7.2903e-01]])
    tensor([[ 1.6507e-04,  4.4403e-01],
            [-3.4622e-03,  7.2903e-01]], grad_fn=<DivBackward0>)


The results from the autograd's `z.backward()`yields the same result as the differentiation using chain rule from Calculus.

8f75aac42786164895b657d5cc0df8ddb09d39e9
ipynb
Jupyter Notebook
Code for paper.ipynb
brianAsimba/Moon
8bc6a745c5ada85f4636a3de54bb12bc043e923a
MIT
Code for paper.ipynb
brianAsimba/Moon
8bc6a745c5ada85f4636a3de54bb12bc043e923a
MIT
Code for paper.ipynb
brianAsimba/Moon
8bc6a745c5ada85f4636a3de54bb12bc043e923a
MIT
2019-04-26T08:35:17.000Z
2019-04-26T08:35:17.000Z
Qwen/Qwen-72B
1. YES
2. YES