# For today's code challenge you will be reviewing yesterdays lecture material. Have fun!

### if you get done early check out [these videos](https://www.3blue1brown.com/neural-networks).

# The Perceptron

The first and simplest kind of neural network that we could talk about is the perceptron. A perceptron is just a single node or neuron of a neural network with nothing else. It can take any number of inputs and spit out an output. What a neuron does is it takes each of the input values, multplies each of them by a weight, sums all of these products up, and then passes the sum through what is called an "activation function" the result of which is the final value.

I really like figure 2.1 found in this [pdf](http://www.uta.fi/sis/tie/neuro/index/Neurocomputing2.pdf) even though it doesn't have bias term represented there.



If we were to write what is happening in some verbose mathematical notation, it might look something like this:

\begin{align}
 y = sigmoid(\sum(weight_{1}input_{1} + weight_{2}input_{2} + weight_{3}input_{3}) + bias)
\end{align}

Understanding what happens with a single neuron is important because this is the same pattern that will take place for all of our networks. 

When imagining a neural network I like to think about the arrows as representing the weights, like a wire that has a certain amount of resistance and only lets a certain amount of current through. And I like to think about the node itselef as containing the prescribed activation function that neuron will use to decide how much signal to pass onto the next layer.

# Activation Functions (transfer functions)

In Neural Networks, each node has an activation function. Each node in a given layer typically has the same activation function. These activation functions are the biggest piece of neural networks that have been inspired by actual biology. The activation function decides whether a cell "fires" or not. Sometimes it is said that the cell is "activated" or not. In Artificial Neural Networks activation functions decide how much signal to pass onto the next layer. This is why they are sometimes referred to as transfer functions because they determine how much signal is transferred to the next layer.

## Common Activation Functions:



# Implementing a Perceptron from scratch in Python

### Establish training data


```python
import numpy as np

np.random.seed(812)

inputs = np.array([
    [0, 0, 1],
    [1, 1, 1],
    [1, 0, 1],
    [0, 1, 1]
])

correct_outputs = [[0], [1], [1], [0]]
```

### Sigmoid activation function and its derivative for updating weights


```python
def sigmoid(x):
  return 1 / (1 + np.exp(-x))

def sigmoid_derivative(x):
  sx = sigmoid(x)
  return sx * (1 - sx)
```

## Updating weights with derivative of sigmoid function:



### Initialize random weights for our three inputs


```python
weights = 2 * np.random.random((3, 1)) - 1
weights
```




    array([[-0.80108081],
           [ 0.5104585 ],
           [ 0.59516067]])



### Calculate weighted sum of inputs and weights


```python
weighted_sum = np.dot(inputs, weights)
weighted_sum
```




    array([[ 0.59516067],
           [ 0.30453837],
           [-0.20592014],
           [ 1.10561918]])



### Output the activated value for the end of 1 training epoch


```python
activated_output = sigmoid(weighted_sum)
activated_output
```




    array([[0.64454837],
           [0.57555158],
           [0.44870111],
           [0.75131149]])



### take difference of output and true values to calculate error


```python
error = correct_outputs - activated_output
error
```




    array([[-0.64454837],
           [ 0.42444842],
           [ 0.55129889],
           [-0.75131149]])




```python
adjustments = error * sigmoid_derivative(activated_output)
adjustments
```




    array([[-0.14549539],
           [ 0.09778778],
           [ 0.13111388],
           [-0.16363003]])




```python
weights += np.dot(inputs.T, adjustments)
weights
```




    array([[-0.57217915],
           [ 0.44461626],
           [ 0.51493691]])



### Put it all together


```python
for iteration in range(10000):
  
  # Weighted sum of inputs/weights
  weighted_sum = np.dot(inputs, weights)
  
  # Activate!
  activated_output = sigmoid(weighted_sum)
  
  # Calculate error
  error = correct_outputs - activated_output
  
  # Adjustments
  adjustments = error * sigmoid_derivative(activated_output)
  
  # Update weights
  weights += np.dot(inputs.T, adjustments)
  
print("Weights after training")
print(weights)

print("Output after training")
print(activated_output)
```

    Weights after training
    [[15.03729668]
     [-0.40682518]
     [-7.23231091]]
    Output after training
    [[7.22398796e-04]
     [9.99387935e-01]
     [9.99592428e-01]
     [4.81060734e-04]]


2207293169d32e524a37d0e975e508dab5f06581
ipynb
Jupyter Notebook
Tuesday Solution.ipynb
eyvonne/Neural_network_foundations_code_challenges
fa9ef104cb4e5dc43e3b6649659778ce10ff17b3
MIT
Tuesday Solution.ipynb
eyvonne/Neural_network_foundations_code_challenges
fa9ef104cb4e5dc43e3b6649659778ce10ff17b3
MIT
Tuesday Solution.ipynb
eyvonne/Neural_network_foundations_code_challenges
fa9ef104cb4e5dc43e3b6649659778ce10ff17b3
MIT
2019-11-05T16:41:55.000Z
2019-11-05T17:09:13.000Z
Qwen/Qwen-72B
1. YES
2. YES