# Introduction to Neural Networks 



## The Perceptron


To get an intuitive idea about Neural Networks, we will code an elementary perceptron. In this example we will illustrate some of the concepts you have just seen, build a small perceptron and make a link between Perceptron and linear classifier.

### Generating some data

Before working with the MNIST dataset, you'll first test your perceptron implementation on a "toy" dataset with just a few data points. This allows you to test your implementations with data you can easily inspect and visualise without getting lost in the complexities of the dataset itself.


Start by loading two basic libraries: `matplotlib` and numpy




```python
# Load the libraries ...

```

Then let us generate some points in a 2D space that will form our dataset (you can add points later if you'd like)


```python
crosses = np.array([[0.5, 1.0], [1.0, 1.5], [1.5, 1.5], [2.0, 1.2], [3.0, 1.7], [1.5, 1.1],[2.1, 1.7]])
circles = np.array([[3.0, 0.5], [4.0, 1.0], [5.0, 0.7], [4.0, 0.2], [5.1, 0.3], [4.2, 0.7]])
```

### Visualising the data

Using `matploblib`, you can display the crosses as crosses (use `marker='x'`) and the circles as circles (use `marker='o'`). You will need to specify that you don't want a line using `linestyle='none'`. You can observe that the points are very easily separable. 


```python
# add your code here to visualise the points

```

### Computing the output of a Perceptron


Let us consider the problem of building a classifier that for a given **new** point will return whether it belongs to the crosses (class 1) or circles (class 0). So for example it would take `(2, 1.5)` and return `1`. 

Define a function `outPerceptron` which takes a 2d vector `x`, a 2d weight vector `w` and a bias `b` and returns the output following the step rule:

$$
\text{output} = \left\{\begin{align} 1\,\, &\text{if}\,\, \langle x, w\rangle -b \, >\,0 \\ 0\,\, &\text{otherwise}\end{align}\right.
$$


```python
# add your code here...

```

You can then enrich the function so that it can take a **sequence of inputs** (in the form of a matrix where each line of the matrix is one input vector) and return the corresponding **sequence of outputs**. 

One way of doing this is to loop over the rows of `X` and for each of them, use the function `outPerceptron` that you just wrote. Store the results in an array `outputs` and return that. Call that function `multiOutPerceptron`/

Once you have that, you can try optimising the function by using a matrix-vector product; call the resulting function `multiOutPerceptron2` (and make sure it leads to the same results!)


```python
# add your code here to implement multiOutPerceptron

# (bonus) add your code here to implement multiOutPerceptron2

```

## = checkpoint 1 =

here, you should copy-paste the following code. If it returns `True` you're good to go on.

```python
np.random.seed(1234)
X = np.random.randn(10, 5)
w = np.random.randn(5)
b = np.random.randn()
np.all(multiOutPerceptron2(X, w, b) == np.array([ 1.,  1.,  0.,  0.,  0.,  1.,  0.,  0.,  1.,  0.]))
```


### Trying different weights and biases

You now have a method that can compute the outputs predicted by an **untrained** perceptron. Can you try picking different weights and biases and see how well you can classify the crosses and circles? 

**Note**: to join the crosses and circles into one `instances` matrix, you can use `np.concatenate((crosses, circles), axis=0)`.

You can maybe start with `w=[1, 1]` and `b=1` and output the result of `multiOutPerceptron`. What is your analysis?


```python
# your code here...

```

With the suggested weights and biases (`([1, 1],1)`), you should see something like 

> `[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]` 

which is clearly not great! Now try with `w=[-0.5, 1]` and `b=-0.2`, what do you observe? 


```python
# your code here...

```

### How did we get there?

This is much better (100% correct on the training data). 
To obtain these values, we found a **separating hyperplane** (here a line) between the points. 
The equation of the line is 

$ y = 0.5x-0.2 $


**Quiz**
- **Can you explain why this line corresponds to the weights and bias we used?**
- **Is this separating line unique? does it matter?**

### Illustrating the output of the Perceptron and the separating line

Copy-paste your code to visualise the crosses and circles above and overlay the separating line in red. 

Can you modify the parameters of the line a little bit and still find a separating line that "works"? 


```python
# your code here..

```

### Testing a few new points

Can you add the following `testPoints` on the plot and discuss what happens to them? 


```python
testPoints = np.array([[1, 0.5],[5, 1.5],[3, 1.1]])
```


```python
# your code here to visualise the situation

```

## = checkpoint 2 =


# Gradient Descent (remember?)

## Coding a simple gradient descent

### Considering some function

Let's consider the following arbitrary function and its gradient

$f(x) = \exp(-\sin(x))x^2$

$f'(x) = -x \exp(-\sin(x)) (x\cos(x)-2)$

It is convenient to define python functions which return the value of the function and its gradient at an arbitrary point $x$. Can you define a function `function` and a function `gradient`? 

* use `np.exp`, `np.sin`, `np.cos` and remember that `x**2` is the squared of `x`


```python
# your code here...

```

### Visualising the function

Can you write a simple code that shows what the function looks like over the interval `[-10,10]`? use at least `100` points in order to have a high enough definition of the line.


```python
# your code here...

```

### Implementing a simple GD

Now let us implement a simple Gradient Descent that uses constant stepsizes. Define two functions:

1. simplest version which doesn't store the intermediate steps that are taken. 
2. a version which does store the steps (useful to visualize what is going on and explain some of the typical behaviour of GD).

Let's call them `simpleGD` and `simpleGD2`. The parameters of both functions will be the initial point `x0`, the stepsize, and the number of steps to be taken.


```python
# your code for simpleGD and simpleGD2

```

### Testing different situations

Try your algorithm `simpleGD` in the following cases:

* $x_0=1, \delta=0.1, n=100$
* $x_0=6, \delta=0.1, n=100$
* $x_0=8, \delta=0.01, n=100$

Can you discuss the results you obtained by having a look at the plot of the function? 

### Visualising the cases

We suggest below a function `viz` which shows the path taken by the gradient descent when computed using `simpleGD2`. 

Use it in the different cases above in order to see what the Gradient Descent does. Try to interpret the different cases.


```python
def viz(x, a=-10, b=10):
    xx  = np.linspace(a, b, 100)
    yy  = function(xx)
    ygd = function(x)
    plt.plot(xx, yy)
    plt.plot(x, ygd, color='red')
    plt.plot(x[0], ygd[0], marker='o', color='green', markersize=10)
    plt.plot(x[len(x)-1], ygd[len(x)-1], marker='o', color='red', markersize=10)
    plt.show()

```


```python
# your code to visualise the different situations here

```

03293ff0c1fc8460f5391c627f37718299a59a04
ipynb
Jupyter Notebook
ads8-neuralnets_content_students/d1-introNN-part1_skeleton.ipynb
JamesOwers/camsparknn
c94b6c8307ff85c819f8efc1f57ab58242bdd7cd
MIT
2021-06-04T14:26:30.000Z
2021-06-04T14:26:30.000Z
ads8-neuralnets_content_tutors/d1-introNN-part1_skeleton.ipynb
JamesOwers/camsparknn
c94b6c8307ff85c819f8efc1f57ab58242bdd7cd
MIT
ads8-neuralnets_content_tutors/d1-introNN-part1_skeleton.ipynb
JamesOwers/camsparknn
c94b6c8307ff85c819f8efc1f57ab58242bdd7cd
MIT
Qwen/Qwen-72B
1. YES
2. YES