# Network Architecture

**DIVE into Deep Learning**
___


```python
%matplotlib inline
from util import *
```

## Mathematical Definition


```python
%%html

```

As shown below, a neural network is organized into layers of computation units called the *neurons*.$\def\abs#1{\left\lvert #1 \right\rvert}
\def\Set#1{\left\{ #1 \right\}}
\def\mc#1{\mathcal{#1}}
\def\M#1{\boldsymbol{#1}}
\def\R#1{\mathsf{#1}}
\def\RM#1{\boldsymbol{\mathsf{#1}}}
\def\op#1{\operatorname{#1}}
\def\E{\op{E}}
\def\d{\mathrm{\mathstrut d}}
$

For $\ell\in \{0,\dots,L\}$ and integer $L\geq 1$, let 
- $\M{a}^{(\ell)}$ be the output of the $\ell$-th layer of the neural network, and
- $a^{(\ell)}_i$ be the $i$-th element of $\M{a}^{(\ell)}$. The element is computed from the output $\M{a}^{(\ell-1)}$ of its previous layer except for $\ell=0$.

The $0$-th layer is called the *input layer*, i.e.,
$$\M{a}^{(0)}:=\M{x}.$$

The $L$-th layer $\M{a}^{(L)}$ is called the *output layer*. All other layers are called the *hidden layers*.

**What should be the neural network output?**

The goal is to train a *classifier* that predicts a label $\R{y}$ for an input feature $\RM{x}$:

- Instead of a hard-decision classifier is a function $f:\mc{X}\to \mc{Y}$ such that
$f(\RM{x})$ predicts $\R{y}$,

- we train a probabilistic classifier $q_{\R{y}|\RM{x}}$ that estimates $p_{\R{y}|\RM{x}}$, i.e.,

$$
\begin{align}
[q_{\R{y}|\RM{x}}(y|\M{x})]_{y\in \mc{Y}} &:= \M{a}^{(L)}.
\end{align}
$$

For MNIST, the goal is to classify the digit type of a handwritten digit.  When given a handwritten digit,
- a hard-decision classifier returns a digit type, and
- a probabilistic classifier returns a distribution of the digit types.

**Why train a probabilistic classifier?**

- A probabilistic classifer is more general and can give a hard decision as well   

  $$f(\RM{x}):=\arg\max_{y\in \mc{Y}} q_{\R{y}|\RM{x}}(y|\RM{x})$$ 
  by returning the estimated most likely digit type.

- A neural network can model the distribution $p_{\R{y}|\RM{x}}(\cdot|\RM{x})$ better than $\R{y}$ because its output is continuous.

**How to ensure $\M{a}^{(L)}$ is a valid probability vector?**

The soft-max activation function is often used for the last layer:

$$ \sigma^{(L)}\left(\left[\begin{smallmatrix}z^{(\ell)}_1 \\ \vdots \\ z^{(\ell)}_k\end{smallmatrix}\right]\right) := \frac{1}{\sum_{i=1}^k e^{z^{(\ell)}_i}} \left[\begin{smallmatrix}e^{z^{(\ell)}_1} \\ \vdots \\ e^{z^{(\ell)}_k}\end{smallmatrix}\right] $$ 

where $k:=\abs{\mc{Y}}=10$ is the number of distinct class labels.

It follows that: 

$$\sum_{i=1}^k a_i^{(L)}  = 1\kern1em \text{and} \kern1em a_i^{(L)}\geq 0\qquad \forall i\in \{1,\dots,k\}.$$

**How are the different layers related?**

$$
\begin{align}
\M{a}^{(\ell)}&:=\begin{cases}
\M{x} & \ell=0\\
\sigma^{(\ell)}(\overbrace{\M{W}^{(\ell)}\M{a}^{(\ell-1)}+\M{b}^{(\ell)}}^{\RM{z}^{(\ell)}:=})& \ell>0;
\end{cases}\tag{net}
\end{align}
$$

- $\M{W}^{(\ell)}$ is a matrix of weights;
- $\M{b}^{(\ell)}$ is a vector called bias; and
- $\sigma^{(\ell)}$ is a reveal-valued function called the *activation function*.

The activation functions $\sigma^{(\ell)}$ for other layers $1\leq \ell<L$ is often the vectorized version of 
-  sigmoid:  

  $$\sigma_{\text{sigmoid}}(z) = \frac{1}{1+e^{-z}}$$
-  rectified linear unit (ReLU): 

  $$ \sigma_{\text{ReLU}}(z) = \max\{0,z\}. $$


```python
%%html
<b>Activation function:</b><br>

```

The following plots the ReLU activation function.


```python
def ReLU(z):
    return np.max([np.zeros(z.shape), z], axis=0)


z = np.linspace(-5, 5, 100)
plt.plot(z, ReLU(z))
plt.xlim(-5, 5)
plt.title(r'ReLU: $\max\{0,z\}$')
plt.xlabel(r'$z$')
plt.show()
```

**Exercise** Complete the vectorized function `sigmoid` using the vectorized exponentiation `np.exp`.


```python
def sigmoid(z):
    # YOUR CODE HERE
    raise NotImplementedError()


plt.plot(z, ReLU(z))
plt.xlim(-5, 5)
plt.title(r'Sigmoid function: $\frac{1}{1+e^{-z}}$')
plt.xlabel(r'$z$')
plt.show()
```


```python
# tests
```

## Implementation

The following uses the [`keras`](https://keras.io/) library to define the basic neural network achitecture.

`keras` runs on top of `tensorflow` and offers a higher-level abstraction to simplify the construction and training of a neural network. ([`tflearn`](https://github.com/tflearn/tflearn) is another library that provides a higher-level API for `tensorflow`.)


```python
def create_simple_model():
    tf.keras.backend.clear_session() # clear keras cache. 
                        # See https://github.com/keras-team/keras/issues/7294
    model = tf.keras.models.Sequential([
        tf.keras.layers.Flatten(input_shape=(28, 28, 1)),
        tf.keras.layers.Dense(16, activation=tf.keras.activations.relu),
        tf.keras.layers.Dense(16, activation=tf.keras.activations.relu),
        tf.keras.layers.Dense(10, activation=tf.keras.activations.softmax)
    ], 'Simple_sequential')
    return model


model = create_simple_model()
model.summary()
```

The above defines [a linear stack](https://www.tensorflow.org/api_docs/python/tf/keras/layers) of [fully-connected layers](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense) after [flattening the input](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Flatten). The method `summary` is useful for [debugging in Keras](https://keras.io/examples/keras_recipes/debugging_tips/).

**Execise** Assign to `n_hidden_layers` the number of hidden layers for the above simple sequential model. 

*Hint:* The layer `Flatten` does not count as a hidden layer since it simply reshapes the input without using any trainable parameters. The output layer also does not count as a hidden layer since its output is the output of the neural network, not intermediate (hidden) values that require further processing by the neurons.


```python
# YOUR CODE HERE
raise NotImplementedError()
n_hidden_layers
```


```python
# tests
```

0c0412c03278dede338b08dffaf9a2a10b039c4d
ipynb
Jupyter Notebook
part2/network.ipynb
ccha23/divedeep
dd9c5e0a589613fa37c467b7863e58c5d22d8d3f
MIT
2021-06-29T00:46:39.000Z
2021-06-29T00:46:39.000Z
part2/network.ipynb
ccha23/divedeep
dd9c5e0a589613fa37c467b7863e58c5d22d8d3f
MIT
part2/network.ipynb
ccha23/divedeep
dd9c5e0a589613fa37c467b7863e58c5d22d8d3f
MIT
2021-07-03T02:44:06.000Z
2021-07-03T02:44:06.000Z
Qwen/Qwen-72B
1. YES
2. YES