# Meta

Filename:  BasicNeuralNets.ipynb

Purpose:  To get an introductory understanding of artificial neural networks in the context of image-recognition.  To illustrate their structure, function, and effectiveness.  In practice, people generally start studying this area using the MNIST data set to construct a neural net to distinguish and correctly label handwritten digits.  This is the traditional "hello world" project in this field, so let's do that.  Let's construct basic digit-recognizing networks (in the form of FFSNNs) using gradient descent and backpropagation to improve performance from a random starting point (that is, from randomly selected initial parameters).

Abbreviation: MNIST = Modified (handwritten digits data set from the U.S.) National Institute of Standards and Technology

Abbreviation: FFSNN = Feed-Forward Sigmoid-Neuron Network

Overview: Watch this video series to get an excellent (and beautiful) overview of how neural networks (such as the one ones built here) are built and function: Neural Networks, by 3Blue1Brown ( https://www.youtube.com/watch?v=aircAruvnKk&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi )

Reference: Much of the strategy and code here is taken from Michael Nielsen's online book on "Neural Networks and Deep Learning" ( http://neuralnetworksanddeeplearning.com ).

# Network IO

The input data set originates from what's called the MNIST database of handwritten digits (found here: http://yann.lecun.com/exdb/mnist/).  This is a set of 70,000 images of hand-written digits, scanned and digitized, each image processed so the digit is size-normalized, centered, and anti-aliased, with 28x28 (=784) pixels, each pixel having a greyscale value (between 0 and 1? -- take a look at one original image example).

Originally split into a training set of 60,000 images and a test set of 10,000 images, we'll split it slightly differently into three groups: training images (for setting parameters), validation images (for setting hyperparameters), and test images (for testing the effectiveness of the resulting network).
\begin{align}
I   &=  \text{set of 70,000 input images}  \\
I1  &=  \text{set of } n1 \text{ (50,000) training images}  \\
I2  &=  \text{set of } n2 \text{ (10,000) validation images}  \\
I3  &=  \text{set of } n3 \text{ (10,000) test images}
\end{align}




# Network Specs

Each network can be visually represented as being composed of nodes and connections, which symbolize the algorithmic relationships between the input data and output data, via intermediate calculations that produce the output from the input.

The nodes (or artificial "neurons") are organized by layers, and each adjacent layer's nodes are all connected to each other.  In other words, adjacent layers are fully connected.
\begin{align}
L  &=  \text{number of layers}  \\
\mathbf{N}
   &=  \text{numbers of nodes/neurons in each layer (vector, } L\text{-dimensional, ordered by layer, from input to output)}  \\
   &=  \left< N_1, ..., N_i, ..., N_L \right>  \\
N_1
   &=  \text{number of nodes/neurons in input layer}  \\
N_i
   &=  \text{numbers of nodes/neurons in } i^{\text{th}} \text{ layer}  \\
N_L
   &=  \text{numbers of nodes/neurons in output layer}
\end{align}

Note: For any $i$ where $1<i<L$, the $i^\text{th}$ layer (or layer-$i$) is called "hidden", presumably because it is part of the eventual "black box" final-product network, where the input and output are the (more) visible parts.

Each node/neuron holds/outputs an "activation" (number).
\begin{align}
\mathbf{a}
   &=  \text{activations of nodes/neurons in each layer (vector, } L\text{-dimensional, ordered by layer, from input to output)}  \\
   &=  \left< \mathbf{a}^1, ..., \mathbf{a}^i, ..., \mathbf{a}^L \right>  \\
\mathbf{a}^i
   &=  \text{activations of nodes/neurons in } i^{\text{th}} \text{ layer (vector, } N_i\text{-dimensional, ordered by node, from top to bottom)}  \\
   &=  \left< a^i_1, ..., a^i_j, ..., a^i_{N_i} \right>
\end{align}

Each activation value in the input layer is a function of the input data (for the first node-layer) or a function of the previous layer's activation


```python

```
