# Building a Neural Network From Scratch


```python
import numpy as np
import gzip
import pickle
from fastai import datasets
import torch
```

Let's try to build a neural network from scratch. Roughly speaking we need to do the following:

 - [x] Get the dataset 
 - [x] Implement matrix multiplication
 - [x] Implement ReLU
 - [x] Implement Softmax 
 - [x] Calculate Loss 
 - [ ] Back prop

### Get the dataset


```python
MNIST_URL='http://deeplearning.net/data/mnist/mnist.pkl'
path = datasets.download_data(MNIST_URL, ext='.gz');
with gzip.open(path, 'rb') as f:
    ((x_train, y_train), (x_valid, y_valid), _) = pickle.load(f, encoding='latin-1')
```


```python
print("Training set: ", len(x_train))
print("Validation set: ", len(x_valid))
```

    Training set:  50000
    Validation set:  10000



```python
#Our first input
x1 = x_train[:1]
x1.shape
```




    (1, 784)




```python
#Our first set of weights and biases
weights = np.random.rand(784, 10)
biases = np.zeros(10)
```

## Implement Matrix Multiplication


```python
def matrix_multiply(a, b):
    
    a_rows, a_columns = a.shape
    b_rows, b_columns = b.shape
    
    print(a_rows, a_columns, sep='x')
    print(b_rows, b_columns, sep='x')
    assert(a_columns == b_rows)
    
    result = np.zeros((a_rows, b_columns))
    
    for i in range(0, a_rows):
        for j in range(0, b_columns):
            result[i,j] = (a[i,:] * b[:,j]).sum()
            
    return result

a = np.array([[1,2,1],[0,1,0], [2,3,4]])
b = np.array([[2,5],[6,7],[1,8]])
matrix_multiply(a, b)
```

    3x3
    3x2





    array([[15., 27.],
           [ 6.,  7.],
           [26., 63.]])



## Implement ReLU


```python
def relu(x):
    #If the item is less than zero, return 0. Otherwise return the original value.
    result = (x > 0) * x
    return result

a = np.array([[1,-2,-1],[0,1,0], [2,3,4]])
relu(a)
```




    array([[1, 0, 0],
           [0, 1, 0],
           [2, 3, 4]])



## Implement Softmax

Next we have to implement a `softmax` function to map from the output of our network to a set of probabilities for each class. Softmax is defined as:

\begin{align}
\sigma(\mathbf{z})_j = \frac{e^{z_j}}{\sum_{k=1}^K e^{z_k}}
\end{align}


```python
def softmax(x):
    assert len(x.shape) == 2 #We're only going to operate on batches
    
    numerator = np.exp(x)
    denominator = np.expand_dims(np.sum(np.exp(x), axis=1), axis=1) 
    return numerator / denominator

probabilities = softmax(np.random.rand(3,5))
```


```python
probabilities
```




    array([[0.175308, 0.229286, 0.29765 , 0.171439, 0.126316],
           [0.144358, 0.156612, 0.331245, 0.155997, 0.211788],
           [0.14033 , 0.23684 , 0.14718 , 0.213848, 0.261802]])



## Calculate Loss

Now we need to calculate loss on a given mini-batch of examples. Cross entropy loss is a fairly standard way of calculating loss for multi-class classification problems. It is defined as:

\begin{align}
{H(p,q)=-\sum _{x}p(x)\,\log q(x).\!}
\end{align}

Where:
 - `p(x)` is `1` in the case of a correct class, `0` otherwise
 - `q(x)` is the probability our network has assigned to the current class


```python
fake_preds = np.random.rand(3,5)
print(fake_preds)
```

    [[0.521736 0.541294 0.21993  0.0036   0.828725]
     [0.25187  0.89769  0.233337 0.263168 0.71167 ]
     [0.02052  0.155222 0.326832 0.979635 0.427007]]



```python
fake_y = np.zeros((3,), dtype=np.long)
fake_y[0] = 0
fake_y[1] = 1
fake_y[2] = 1
fake_y
```




    array([0, 1, 1])




```python
#Double check the result against PyTorch
loss = torch.nn.CrossEntropyLoss()
results = loss(torch.tensor(fake_preds), torch.tensor(fake_y, dtype=torch.long))
results.numpy()
```




    array(1.556455)




```python
def cross_entropy_loss(y_pred, y_target):
    # Rescale the raw values as probabilities
    y_pred = softmax(y_pred)
    numberOfItems, numberOfClasses = y_pred.shape    
    totalLoss = np.log(y_pred[range(numberOfItems), y_target]).sum()
    #Return average loss
    return -(totalLoss) / numberOfItems

cross_entropy_loss(fake_preds, fake_y)
```




    1.5564550739075456



## Back Propagation

Now that we've implemented matrix multiplication, ReLU, Softmax and calculated loss, we've got all the parts necessary to complete a single forward pass. Next we'll implement backwards propagation.


```python
def cross_entropy_loss_grad(y_pred, y_target):
    
    batchSize = len(y_target)
    gradients = softmax(y_pred)
    gradients[range(batchSize), y_target] -= 1
    
    gradients = gradients / batchSize
    
    return gradients
```


```python
y_target
```




    array([[0],
           [1],
           [1]])




```python
cross_entropy_loss_grad(fake_preds, fake_y)
```




    array([[-0.262625,  0.072105,  0.052288,  0.042116,  0.096116],
           [ 0.051406, -0.235273,  0.050462,  0.05199 ,  0.081415],
           [ 0.043782, -0.283238,  0.059473,  0.114243,  0.06574 ]])




```python

```

e9491b6ae0347aefc6263a9ead823046ff8726e2
ipynb
Jupyter Notebook
main.ipynb
JoshVarty/NeuralNetFromScratch
ac54baa63ff0cc3b00905dba76f492a4de22394c
MIT
main.ipynb
JoshVarty/NeuralNetFromScratch
ac54baa63ff0cc3b00905dba76f492a4de22394c
MIT
main.ipynb
JoshVarty/NeuralNetFromScratch
ac54baa63ff0cc3b00905dba76f492a4de22394c
MIT
Qwen/Qwen-72B
1. YES
2. YES