# A Review of Yesterday's Content:

- Neural Networks are made up of layers of nodes or neurons. 
- There are three kinds of layers: input, hidden, and output layers.
<center></center>
- Each layer is made up of individual neurons which have a corresponding weight and bias.
<center></center>
- Signal is passed from layer to layer through a network by:
 - Taking in inputs from the training data (or previous layer)
 - Multiplying each input by its corresponding weight (think arrow/connecting line)
 - Adding a bias to this weighted sum of inputs and weights
 - Activating this weighted sum + bias by squishifying it with sigmoid or some other activation function. With a single perceptron with three inputs, calculating the output from the node is done like so:
\begin{align}
 y = sigmoid(\sum(weight_{1}input_{1} + weight_{2}input_{2} + weight_{3}input_{3}) + bias)
\end{align}
 - this final activated value is the signal that gets passed onto the next layer of the network.


## Multilayer Perceptrons AKA Feedforward Neural Networks

Yesterday we talked about the capabilities and limitations of individual perceptrons.

- Perceptrons are like a single neuron or layer of neurons in a neural network
- Perceptrons can only fit linear boundaries between classes

What is a Feedforward Neural Network? Well, it's a Neural Network made up of multiple perceptrons that has at least 1 hidden layer (a layerbetween the input and output layers). When we use the [Tensorflow Playground](https://playground.tensorflow.org) website to explore different neural network architectures, we're really just composing different architectures (topologies) of feedforward neural networks. However, when we give the network multiple hidden layers it should be designated as a "deep" feedforward neural network.

### Deep Feedforward Neural Network
<center></center>

# How do Neural Networks Learn?

## Calculating *"loss"* or *"error"*

We've talked about how in order to evaluate a network's performance, the data is "fed forward" until predictions are obtained and then the "loss" or "error" for a given observation is ascertained by looking at what the network predicted for that observation and comparing it to what it *should* have predicted. 

The error for a given observation is calculated by taking the square of the difference between the predicted value and the actual value. 

We can summarize the overal quality of a network's predictions by finding the average error across all observations. This gives us the "Mean Squared Error." which hopefully is a fairly familiar model evaluation metric by now. Graphing the MSE over each epoch (training cycle) is a common practice with Neural Networks. This is what you're seeing in the top right corner of the Tensorflow Playground website as the number of "epochs" climbs higher and higher.

We also graphed loss over a number of iterations when working with gradient descent. Neural Networks use gradient descent to find their optimal parameters.

<center></center>

## What is an "Epoch"?

An "Epoch" is one cycle of:
* passing our data forward through the network
* measuring error given our specified cost function
* updating weights within our network (via gradient descent) to hopefully improve the quality of our predictions on the next iteration

### Batch vs Minibatch vs Stochastic Gradient Descent Epochs

You may have heard these variations on the training process referenced in the 3Blue1Brown videos about backpropagation. "Minibatch" Gradient Descent means that instead of passing all of our data through the network for a given epoch (Batch GD), we just pass a randomized portion of our data through the network for each epoch. 

**Stochastic Gradient Descent** is when we make updates to our weights after forward propagating each individual training observation.

### Why might we use something like Minibatch GD? 

Neural Networks often require high quantities of training data. Consider the [MNIST Dataset](http://yann.lecun.com/exdb/mnist/):



This dataset is what has been used in the 3Blue1Brown videos for demonstrating important neural network concepts. The dataset has 60,000 training observations and 10,000 test observations. SInce there are 10 numeric digits that works out to be 6,000 training images per digit. Do you think our neural network needs to look at all 6000 number 3s in order to get an idea of how accurate or innacurate it is at recognizing the number 3? No, it doesn't. It's actually pretty computationally expensive to run the entire dataset through the network for each epoch, so we'll just randomly select a portion of the dataset for each training epoch.

How much of the data should you pass through for each epoch? Well, you'll probably get sick of hearing this response to questions like this, but `batch_size` is another hyperparameter for our model that we'll need to tune to try and get the best results. We'll try different amounts out and try to find an optimal "batch size."

### A note about Hyperparameters

Neural Networks have many more hyperparameters than other machine learning algorithms which is part of what makes them a beast to train. 1) You need more data to train them on. 2) They're complex so they take longer to train. 3) They have lots and lots of hyperparameters which we need to find the most optimal combination of, so we might end up training our model dozens or hundreds of times with different combinations of hyperparameters in order to try and squeeze out a few more tenths of a percent of accuracy. 

# Backpropagation

Backpropagation is short for ["Backwards Propagation of errors"](https://en.wikipedia.org/wiki/Backpropagation) and refers to a specific (rather calculus intensive) algorithm for how weights in a neural network are updated in reverse order at the end of each training epoch. Our purpose today is to demonstrate the backpropagation algorithm on a simple Feedforward Neural Network and in so doing help you get a grasp on the main process. If you want to understand all of the underlying calculus of how the gradients are calculated then you'll need to dive into it yourself, [3Blue1Brown's video is a great starting place](https://www.youtube.com/watch?v=tIeHLnjs5U8). Watch this Welch Labs series [Neural Networks Demystified](https://www.youtube.com/watch?v=bxe2T-V8XRs) if you want a rapid yet orderly walkthrough of the main intuitions and math behind the backpropagation algorithm. 

## FeedForward NN for predicting Test Scores



### Generate some Fake Data


```python
import numpy as np
np.random.seed(1)

# hours studying, hours sleeping
X = np.array([
    [2, 9],
    [1, 5],
    [3, 6]
], dtype=float)

# test scores
y = np.array([
    [92],
    [86],
    [89]
], dtype=float)
```


```python
# standardize by feature (divide by max to get relative to maximum)
X = X / np.amax(X, axis=0)
y = y / 100
```


```python
print("Studying, Sleeping\n", X)
print("Test Score\n", y)
```

    Studying, Sleeping
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Test Score
     [[0.92]
     [0.86]
     [0.89]]


### Lets create a Neural_Network class to contain this functionality
* How many random weights do we need to initialize? "Fully-connected Layers"
* Implement Feedforward Functionality
  * After this step our neural network should be able to generate an output even though it has not been trained.


```python
class NeuralNetwork(object):
    def __init__(self):
        self.inputs = 2
        self.hiddenNodes = 3
        self.outputNodes = 1
        # weights for each layer
        self.weights1 = np.random.randn(self.inputs, self.hiddenNodes) # 2x3
        self.weights2 = np.random.randn(self.hiddenNodes, self.outputNodes) # 3x1
        
    def sigmoid(self, s):
        return 1 / (1 + np.exp(-s))
        
    def feed_forward(self, X):
        """Calculate the NN inference using feed forwad."""
        self.hidden_sum = np.dot(X, self.weights1) # 3x3
        self.activated_hidden = self.sigmoid(self.hidden_sum)
        self.output_sum = np.dot(self.activated_hidden, self.weights2)
        self.activated_output = self.sigmoid(self.output_sum)
        return self.activated_output
```


```python
nn = NeuralNetwork()
print("Layer 1 weights: \n", nn.weights1)
print("Layer 2 weights: \n", nn.weights2)
```

    Layer 1 weights: 
     [[ 1.62434536 -0.61175641 -0.52817175]
     [-1.07296862  0.86540763 -2.3015387 ]]
    Layer 2 weights: 
     [[ 1.74481176]
     [-0.7612069 ]
     [ 0.3190391 ]]


### Can we generate an output?


```python
print(X[0])
output = nn.feed_forward(X[0])
print('output ', output)
```

    [0.66666667 1.        ]
    output  [0.60623806]


### Calculating Error


```python
output_all = nn.feed_forward(X)
error_all = y - output_all
print(output_all, error_all, sep='\n')
```

    [[0.60623806]
     [0.6168044 ]
     [0.71212205]]
    [[0.31376194]
     [0.2431956 ]
     [0.17787795]]


## Backpropagation

AKA assigning blame for bad predictions and delivering justice - repeatedly and a little bit at a time.

What in our model could be causing our predictions to suck so bad? 

Well, we know that our inputs (X) and outputs (y) are correct, if they weren't then we would have bigger problems than understanding backpropagation.

We also know that our activation function (sigmoid) is working correctly. It can't be blamed because it just does whatever we tell it to and transforms the data in a known way.

So what are the potential culprits for these terrible predictions? The **weights** of our model. Here's the problem though. I have weights that exist in both layers of my model. How do I know if the weights in the first layer are to blame, or the second layer, or both? 

Lets investigate. And see if we can just eyeball what should be updated.


```python
# Look at how a single observation is processed
print("Inputs", X[0], 
      "Initial Weights", nn.weights1, 
      "Weighted Sum 1", nn.hidden_sum, 
      "Activations from first layer", nn.activated_hidden, 
      "Second Layer Weights", nn.weights2, 
      "Weighted Sum 2", nn.output_sum, 
      "Predictions", output, 
      "Correct Output", y[0], 
      "Error", y[0]-output,
      sep="\n")
```

    Inputs
    [0.66666667 1.        ]
    Initial Weights
    [[ 1.62434536 -0.61175641 -0.52817175]
     [-1.07296862  0.86540763 -2.3015387 ]]
    Weighted Sum 1
    [[ 0.00992829  0.45757002 -2.6536532 ]
     [-0.05464522  0.27686321 -1.45468986]
     [ 0.90903295 -0.03481799 -2.06253088]]
    Activations from first layer
    [[0.50248205 0.61243756 0.0657642 ]
     [0.48634209 0.56877703 0.18928084]
     [0.71280223 0.49129638 0.11279232]]
    Second Layer Weights
    [[ 1.74481176]
     [-0.7612069 ]
     [ 0.3190391 ]]
    Weighted Sum 2
    [[0.43152625]
     [0.47600639]
     [0.90571269]]
    Predictions
    [0.60623806]
    Correct Output
    [0.92]
    Error
    [0.31376194]


Why is my error so big?

My error is so big because my prediction is low.

Why are my prediction low?

Because either:

  1. Second layer **weights** are low  
  2. OR Activations coming from the first layer are low
  
How are activations from the first layer determined? 

  1. By inputs - fixed
  * by **weights** - variable
  
The only thing that I have control over throughout this process in order to increase the value of my final predictions is to either increase weights in layer 2 or increase weights in layer 1. 

Imagine that you could only change your weights by a fixed amount. Say you have .3 and you have to split that up and disperse it over your weights so as to increase your predictions as much as possible. (This isn't actually what happens, but it will help us identify which weights we would benefit the most from moving.)

I need to increase weights of my model somewhere, I'll get the biggest bang for my buck if I increase weights in places where I'm already seeing high activation values -because they end up getting multiplied together before being passed to the sigmoid function. 

> "Neurons that fire together, wire together"


```python
# I want activations that correspond to negative weights to be lower
# and activations that correspond to positive weights to be higher
```

## Diagnosing Backpropagation issues by hand. (not recommended)

Our model has 9 total weights (6 in the first layer, 3 in the last layer) that could be off.

1. Calculate Error for a given each observation
* Does the error indicate that I'm overestimating or underestimating in my prediction?
* Look at final layer weights to get an idea for which weights are helping pass desireable signals and which are stifling desireable signals
* Also go to the previous layer and see what can be done to boost activations that are associated with helpful weights, and limit activations that are associated with unhelpful weights.

### However, we would prefer to do all this in a fancy automated, controlled calculus way.

5. Repeat steps 1-4 for every observation in a given batch, and then given the network's cost function, calculate its gradient using calculus and update weights associated with the (negative) gradient of the cost function. 

Remember that we have 9 weights in our network therefore the gradient that comes from our gradient descent calculation will be the vector that takes us in the most downward direction along some function in 9-dimensional hyperspace.

\begin{align}
C(w1, w2, w3, w4, w5, w6, w7, w8, w9)
\end{align}

You should also know that with neural networks it is common to have gradients that are not convex (like what we saw when we applied gradient descent to linear regression). Due to the high complexity of these models and their nonlinearity, it is common for gradient descent to get stuck in a local minimum, but there are ways to combat this:

1) Stochastic Gradient Descent

2) More advanced Gradient-Descent-based "Optimizers" - See Stretch Goals on assignment.


## Adding BackPropogation to our implementation (non Gradient-Descent)


```python
class Neural_Network(object):
    def __init__(self):
        self.inputs = 2
        self.hiddenNodes = 3
        self.outputNodes = 1

        # Initlize Weights
        self.L1_weights = np.random.randn(self.inputs, self.hiddenNodes) # (3x2)
        self.L2_weights = np.random.randn(self.hiddenNodes, self.outputNodes) # (3x1)

    def feed_forward(self, X):
        # Weighted sum between inputs and hidden layer
        self.hidden_sum = np.dot(X, self.L1_weights)
        # Activations of weighted sum
        self.activated_hidden = self.sigmoid(self.hidden_sum)
        # Weighted sum between hidden and output
        self.output_sum = np.dot(self.activated_hidden, self.L2_weights)
        # final activation of output
        self.activated_output = self.sigmoid(self.output_sum)
        return self.activated_output

    def sigmoid(self, s):
        return 1/(1+np.exp(-s))

    def sigmoidPrime(self, s):
        """derivative of sigmoid"""
        return s * (1 - s)

    def backward(self, X, y, o):
        """backward propgate through the network"""
        self.o_error = y - o # error in output
        self.o_delta = self.o_error * self.sigmoidPrime(o) # applying derivative of sigmoid to error
  
        self.z2_error = self.o_delta.dot(self.L2_weights.T) # z2 error: how much our hidden layer weights contributed to output error
        self.z2_delta = self.z2_error * self.sigmoidPrime(self.activated_hidden) # applying derivative of sigmoid to z2 error
  
        self.L1_weights += X.T.dot(self.z2_delta) # adjusting first set (input --> hidden) weights
        self.L2_weights += self.activated_hidden.T.dot(self.o_delta) # adjusting second set (hidden --> output) weights

    def train(self, X, y):
        o = self.feed_forward(X)
        self.backward(X, y, o)
```


```python
NN = Neural_Network()
for i in range(1000): # trains the NN 1,000 times
    if i+1 in [1,2,3,4,5] or (i+1) % 50 == 0:
        print('+---------- EPOCH', i+1, '-----------+')
        print("Input: \n", X) 
        print("Actual Output: \n", y)  
        print("Predicted Output: \n" + str(NN.feed_forward(X))) 
        print("Loss: \n" + str(np.mean(np.square(y - NN.feed_forward(X))))) # mean sum squared loss
        print("\n")
    NN.train(X, y)
```

    +---------- EPOCH 1 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.52433559]
     [0.51924096]
     [0.51502029]]
    Loss: 
    0.13775894381073592
    
    
    +---------- EPOCH 2 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.61638089]
     [0.60060247]
     [0.60029617]]
    Loss: 
    0.08113331898809754
    
    
    +---------- EPOCH 3 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.67975295]
     [0.65752584]
     [0.66008717]]
    Loss: 
    0.05052477938399689
    
    
    +---------- EPOCH 4 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.72298247]
     [0.69707787]
     [0.70161669]]
    Loss: 
    0.03361593234908083
    
    
    +---------- EPOCH 5 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.75354918]
     [0.72552618]
     [0.73144002]]
    Loss: 
    0.02364344984289771
    
    
    +---------- EPOCH 50 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.89817372]
     [0.8693767 ]
     [0.88023095]]
    Loss: 
    0.00021991447262439465
    
    
    +---------- EPOCH 100 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.90403648]
     [0.87561831]
     [0.88678081]]
    Loss: 
    0.00016970965466925458
    
    
    +---------- EPOCH 150 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.90464459]
     [0.8761624 ]
     [0.88760286]]
    Loss: 
    0.00016758601956212707
    
    
    +---------- EPOCH 200 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.90470047]
     [0.87610621]
     [0.88781732]]
    Loss: 
    0.00016608320206239755
    
    
    +---------- EPOCH 250 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.9046928 ]
     [0.87598343]
     [0.88795802]]
    Loss: 
    0.00016465003512956114
    
    
    +---------- EPOCH 300 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.90467833]
     [0.87585606]
     [0.88808705]]
    Loss: 
    0.00016327587429809022
    
    
    +---------- EPOCH 350 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.9046637 ]
     [0.87573126]
     [0.88821179]]
    Loss: 
    0.00016195754331315595
    
    
    +---------- EPOCH 400 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.90464966]
     [0.87560978]
     [0.88833321]]
    Loss: 
    0.00016069214593036286
    
    
    +---------- EPOCH 450 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.90463627]
     [0.87549157]
     [0.88845152]]
    Loss: 
    0.00015947696134234877
    
    
    +---------- EPOCH 500 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.90462351]
     [0.87537654]
     [0.88856682]]
    Loss: 
    0.0001583094313983074
    
    
    +---------- EPOCH 550 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.90461136]
     [0.87526456]
     [0.88867921]]
    Loss: 
    0.00015718714930283785
    
    
    +---------- EPOCH 600 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.90459979]
     [0.87515553]
     [0.88878879]]
    Loss: 
    0.0001561078491550748
    
    
    +---------- EPOCH 650 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.90458878]
     [0.87504935]
     [0.88889563]]
    Loss: 
    0.00015506939631380884
    
    
    +---------- EPOCH 700 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.9045783 ]
     [0.87494591]
     [0.88899983]]
    Loss: 
    0.0001540697785221398
    
    
    +---------- EPOCH 750 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.90456833]
     [0.87484512]
     [0.88910146]]
    Loss: 
    0.0001531070977244867
    
    
    +---------- EPOCH 800 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.90455885]
     [0.87474688]
     [0.8892006 ]]
    Loss: 
    0.00015217956251462378
    
    
    +---------- EPOCH 850 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.90454985]
     [0.87465113]
     [0.88929732]]
    Loss: 
    0.00015128548115943705
    
    
    +---------- EPOCH 900 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.9045413 ]
     [0.87455776]
     [0.8893917 ]]
    Loss: 
    0.00015042325514852734
    
    
    +---------- EPOCH 950 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.90453319]
     [0.8744667 ]
     [0.8894838 ]]
    Loss: 
    0.00014959137322461912
    
    
    +---------- EPOCH 1000 -----------+
    Input: 
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output: 
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output: 
    [[0.9045255 ]
     [0.87437787]
     [0.88957368]]
    Loss: 
    0.00014878840585406128
    
    


## A true GD-based implementation from [Welch Labs on YouTube](https://www.youtube.com/watch?v=bxe2T-V8XRs)


```python
class Neural_Network(object):
    def __init__(self):        
        #Define Hyperparameters
        self.inputLayerSize = 2
        self.outputLayerSize = 1
        self.hiddenLayerSize = 3
        
        #Weights (parameters)
        self.W1 = np.random.randn(self.inputLayerSize,self.hiddenLayerSize)
        self.W2 = np.random.randn(self.hiddenLayerSize,self.outputLayerSize)
        
    def forward(self, X):
        #Propogate inputs though network
        self.z2 = np.dot(X, self.W1)
        self.a2 = self.sigmoid(self.z2)
        self.z3 = np.dot(self.a2, self.W2)
        yHat = self.sigmoid(self.z3) 
        return yHat
        
    def sigmoid(self, z):
        #Apply sigmoid activation function to scalar, vector, or matrix
        return 1/(1+np.exp(-z))
    
    def sigmoidPrime(self,z):
        #Gradient of sigmoid
        return np.exp(-z)/((1+np.exp(-z))**2)
    
    def costFunction(self, X, y):
        #Compute cost for given X,y, use weights already stored in class.
        self.yHat = self.forward(X)
        J = 0.5*sum((y-self.yHat)**2)
        return J
        
    def costFunctionPrime(self, X, y):
        #Compute derivative with respect to W and W2 for a given X and y:
        self.yHat = self.forward(X)
        
        delta3 = np.multiply(-(y-self.yHat), self.sigmoidPrime(self.z3))
        dJdW2 = np.dot(self.a2.T, delta3)
        
        delta2 = np.dot(delta3, self.W2.T)*self.sigmoidPrime(self.z2)
        dJdW1 = np.dot(X.T, delta2)  
        
        return dJdW1, dJdW2
    
    #Helper Functions for interacting with other classes:
    def getParams(self):
        #Get W1 and W2 unrolled into vector:
        params = np.concatenate((self.W1.ravel(), self.W2.ravel()))
        return params
    
    def setParams(self, params):
        #Set W1 and W2 using single paramater vector.
        W1_start = 0
        W1_end = self.hiddenLayerSize * self.inputLayerSize
        self.W1 = np.reshape(params[W1_start:W1_end], (self.inputLayerSize , self.hiddenLayerSize))
        W2_end = W1_end + self.hiddenLayerSize*self.outputLayerSize
        self.W2 = np.reshape(params[W1_end:W2_end], (self.hiddenLayerSize, self.outputLayerSize))
        
    def computeGradients(self, X, y):
        dJdW1, dJdW2 = self.costFunctionPrime(X, y)
        return np.concatenate((dJdW1.ravel(), dJdW2.ravel()))
```


```python
from scipy import optimize

class trainer(object):
    def __init__(self, N):
        #Make Local reference to network:
        self.N = N
        
    def callbackF(self, params):
        self.N.setParams(params)
        self.J.append(self.N.costFunction(self.X, self.y))   
        
    def costFunctionWrapper(self, params, X, y):
        self.N.setParams(params)
        cost = self.N.costFunction(X, y)
        grad = self.N.computeGradients(X,y)
        
        return cost, grad
        
    def train(self, X, y):
        #Make an internal variable for the callback function:
        self.X = X
        self.y = y

        #Make empty list to store costs:
        self.J = []
        
        params0 = self.N.getParams()

        options = {'maxiter': 200, 'disp' : True}
        _res = optimize.minimize(self.costFunctionWrapper, params0, jac=True, method='BFGS', \
                                 args=(X, y), options=options, callback=self.callbackF)

        self.N.setParams(_res.x)
        self.optimizationResults = _res
```


```python
NN = Neural_Network()
```


```python
T = trainer(NN)
```


```python
T.train(X,y)
```

    Optimization terminated successfully.
             Current function value: 0.000000
             Iterations: 73
             Function evaluations: 81
             Gradient evaluations: 81



```python
print("Predicted Output: \n" + str(NN.forward(X))) 
print("Loss: \n" + str(np.mean(np.square(y - NN.forward(X))))) # mean sum squared loss
```

    Predicted Output: 
    [[0.92001045]
     [0.85999675]
     [0.88999721]]
    Loss: 
    4.2507090512976365e-11



```python
import matplotlib.pyplot as plt
plt.plot(T.J)
plt.xlabel('Iterations')
plt.ylabel('Cost')
plt.show();
```

24703d751bf0ba2542b98c4f17d122c90659e254
ipynb
Jupyter Notebook
15-Neural-Networks/02_Backprop.ipynb
shalevy1/data-science-journal
2a6beaf5bf328e257b638a695983457a9f3cd7ce
MIT
2019-03-05T04:44:48.000Z
2022-03-24T09:47:48.000Z
15-Neural-Networks/02_Backprop.ipynb
emichris/data-science-journal
2a6beaf5bf328e257b638a695983457a9f3cd7ce
MIT
15-Neural-Networks/02_Backprop.ipynb
emichris/data-science-journal
2a6beaf5bf328e257b638a695983457a9f3cd7ce
MIT
2019-03-07T05:08:03.000Z
2022-01-05T11:32:51.000Z
Qwen/Qwen-72B
1. YES
2. YES