```python
#Author-Vishal Burman
```

## Vectorization and mini-batch


```python
# H1=sigma(W1*X+b1)
# H2=sigma(W2*H1+b2)
# o=softmax(W3*H2+b3)
```


```python
%matplotlib inline
from mxnet import autograd, nd
```


```python
import matplotlib.pyplot as plt
```

## Relu Function


```python
# ReLU(z)= max(z, 0)
```


```python
# Retains only positive elements and discards negative elements
```


```python
x=nd.arange(-8.0, 8.0, 0.1)
x.attach_grad()
```


```python
with autograd.record():
    y=x.relu()
# plt.figure(figsize=(4, 2.5))
# plt.plot(x, y, 'x', 'relu(x)')
plt.plot(x.asnumpy(), y.asnumpy())
```


```python
# When input is negative the derivative of the ReLU function is zero
# When the input is positive the derivative of the ReLU function is one
# When the input is zero the ReLU function is not differentiable and we take left derivative as 0
```


```python
y.backward()
```


```python
plt.plot(x.asnumpy(), x.grad.asnumpy(), 'x', 'grad of ReLU')
```


```python
# The reason for using the ReLU is that its derivatives are particularly well behaved
# They vanish or just let the argument through
# This makes argument better behaved and and reduces the vanishing gradient problem
```

## Sigmoid Function


```python
#sigmoid(x)=1/(1+exp(-x))
# The sigmoid function takes the value from R and transforms it into range(0, 1)
```


```python
with autograd.record():
    y=x.sigmoid()
plt.plot(x.asnumpy(), y.asnumpy(), 'x', 'sigmoid(x)')
```


```python
# The derivative of the sigmoid function is given by:
```

\begin{equation}
\frac{d}{dx} \mathrm{sigmoid}(x) = \frac{\exp(-x)}{(1 + \exp(-x))^2} = \mathrm{sigmoid}(x)\left(1-\mathrm{sigmoid}(x)\right).
\end{equation}


```python
y.backward()
```


```python
plt.plot(x.asnumpy(),x.grad.asnumpy(), 'x', 'grad of sigmoid')
```


```python
# When the input is 0, the derivative of the sigmoid function reaches a max of 0.25
# As input diverges from 0 the derivative approaches 0
```

## Tanh function


```python
# Like the sigmoid function the tanh function squashes its input and transforms the element into the interval of
# (-1, 1)
```

\begin{equation}
\text{tanh}(x) = \frac{1 - \exp(-2x)}{1 + \exp(-2x)}.
\end{equation}


```python
with autograd.record():
    y=x.tanh()
plt.plot(x.asnumpy(), y.asnumpy(), 'x', 'tanh(x)')
```


```python
# When the input nears 0 the function approaches a linear transformation
# The shape is similar to sigmoid function but tanh function exhibits point symmetry
```


```python
# The derivative of the tanh function is:
```

\begin{equation}
\frac{d}{dx} \mathrm{tanh}(x) = 1 - \mathrm{tanh}^2(x).
\end{equation}


```python
# As the input approaches 0 the derivative reaches a max of 1
# As the input diverges from 0 the derivative approaches 0
```


```python

```

ed402076d000e04421b10695d5d98cdc1d56c569
ipynb
Jupyter Notebook
Multilayer_NN/test1.ipynb
vishal-burman/MXNet_Architectures
d4e371226e814c1507974244c4642b906566f1d8
MIT
Multilayer_NN/test1.ipynb
vishal-burman/MXNet_Architectures
d4e371226e814c1507974244c4642b906566f1d8
MIT
2020-03-24T17:14:05.000Z
2021-02-02T22:01:48.000Z
Multilayer_NN/test1.ipynb
vishal-burman/MXNet_Architectures
d4e371226e814c1507974244c4642b906566f1d8
MIT
Qwen/Qwen-72B
1. YES
2. YES