## Problem Set 13

First the exercises:
* The exercise did not specify a loss function, but let us assume the standard average cross entropy function. To simplify things let us not use biases. Then the logistic regression model involves the following functions:
\begin{align}
f(x,W_1) &= xW_1 \\
g(f) & = e^f \textrm{(pointwise)} \\
p(g) &= \sum \pi_i g \textrm{ (sum across all vectors in sample)} \\
h(g, p) &= g/p \textrm{ (applied sample wise)} \\
logp(p) &= log(p+epsilon) \textrm{ (epsilon is some small constant to avoid underflow)}\\
sub(f,logp) &= f-logp \textrm{ (broadcast) } \\
pl(y,sub) &= y \cdot sub \textrm{ (Hadamard product)} \\
l(p1) &= mean(pl) \textrm{ (samplewise)}\\
al(pl) &=mean(pl) 
\end{align}

Here $h$ yields the predictions, $l$ yields the samplewise cross entropy, $al$ yields the average cross entropy.

Now we have to figure out the weight update rule. For this we need to calculate $\frac{dal}{dW_1}$ by the chain rule.

Here is a simplified form
\begin{align}
 \frac{dal}{dW_1} &= \frac{dal}{dl}(\frac{dl}{dsub} \left(\frac{dsub}{df}\frac{df}{dW_1}+\frac{dsub}{dlogp}\frac{dlogp}{dW_1}\right)\\
 \frac{df}{dW_1}_{(s,a,b,c,d)} &= x_{(s,a,c)}\delta(b==d) \\
 \frac{dlogp}{dW_1} &=\frac{dlogp}{dp}\frac{dp}{dg}\frac{dg}{df}\frac{df}{dW_1}\\
 \frac{dg}{df}_{(s,a,b,c)} &= e^f \delta(s==b)\delta(a==c) \\
 \frac{dp}{dg}_{(s,a,b)} &= \delta(s==b)
\end{align}

### Python Lab

Now let us load our standard libraries.


```python
import numpy as np
import pandas as pd
import keras
from keras.datasets import mnist
```

    Using TensorFlow backend.


Let us load up a sample dataset.


```python
(x_train, y_train), (x_test, y_test) = mnist.load_data()
```


```python
x_train.shape
```




    (60000, 28, 28)




```python
import matplotlib.pyplot as plt
```


```python
%matplotlib inline
```


```python
randix = np.random.randint(0,60000)
plt.imshow(x_train[randix])
print("Label is {}.".format(y_train[randix]))
```


```python
x_train_f = x_train.reshape(60000,-1)
```


```python
x_train_f.shape
```




    (60000, 784)




```python
x_test_f = x_test.reshape(-1, 28**2)
```


```python
x_test_f.shape
```




    (10000, 784)




```python
from sklearn.preprocessing import StandardScaler
```


```python
scaler = StandardScaler()
```


```python
scaler.fit(x_train_f)
```

    /Users/justin/anaconda/envs/py36/lib/python3.6/site-packages/sklearn/utils/validation.py:429: DataConversionWarning: Data with input dtype uint8 was converted to float64 by StandardScaler.
      warnings.warn(msg, _DataConversionWarning)





    StandardScaler(copy=True, with_mean=True, with_std=True)




```python
train_x_tr = scaler.transform(x_train_f)
```

    /Users/justin/anaconda/envs/py36/lib/python3.6/site-packages/sklearn/utils/validation.py:429: DataConversionWarning: Data with input dtype uint8 was converted to float64 by StandardScaler.
      warnings.warn(msg, _DataConversionWarning)



```python
test_x_tr = scaler.transform(x_test_f)
```

    /Users/justin/anaconda/envs/py36/lib/python3.6/site-packages/sklearn/utils/validation.py:429: DataConversionWarning: Data with input dtype uint8 was converted to float64 by StandardScaler.
      warnings.warn(msg, _DataConversionWarning)



```python
from sklearn.preprocessing import OneHotEncoder as OHE
```


```python
ohe = OHE(sparse = False)
y_train_ohe = ohe.fit_transform(y_train.reshape(-1,1))
y_test_ohe = ohe.fit_transform(y_test.reshape(-1,1))
```


```python
np.argmax(y_train_ohe[randix]) == y_train[randix]
```




    True




```python
from keras.models import Sequential
from keras.layers import Dense, Dropout, BatchNormalization
```


```python
logreg = Sequential()
```


```python
logreg.add(Dense(10,activation='softmax',input_dim = (28**2)))
```


```python
from keras.optimizers import Adam, SGD
```


```python
logreg.compile(optimizer=Adam(),
             loss='categorical_crossentropy',
             metrics=['accuracy'])
```


```python
history=logreg.fit(train_x_tr, y_train_ohe, validation_split=0.05,
                 epochs=25, batch_size=64)
```

    Train on 57000 samples, validate on 3000 samples
    Epoch 1/25
    57000/57000 [==============================] - 1s - loss: 0.4622 - acc: 0.8704 - val_loss: 0.2408 - val_acc: 0.9353
    Epoch 2/25
    57000/57000 [==============================] - 1s - loss: 0.3149 - acc: 0.9145 - val_loss: 0.2305 - val_acc: 0.9420
    Epoch 3/25
    57000/57000 [==============================] - 1s - loss: 0.2950 - acc: 0.9206 - val_loss: 0.2324 - val_acc: 0.9420
    Epoch 4/25
    57000/57000 [==============================] - 1s - loss: 0.2844 - acc: 0.9229 - val_loss: 0.2295 - val_acc: 0.9430
    Epoch 5/25
    57000/57000 [==============================] - 1s - loss: 0.2779 - acc: 0.9256 - val_loss: 0.2262 - val_acc: 0.9433
    Epoch 6/25
    57000/57000 [==============================] - 1s - loss: 0.2723 - acc: 0.9274 - val_loss: 0.2285 - val_acc: 0.9450
    Epoch 7/25
    57000/57000 [==============================] - 1s - loss: 0.2681 - acc: 0.9283 - val_loss: 0.2306 - val_acc: 0.9447
    Epoch 8/25
    57000/57000 [==============================] - 1s - loss: 0.2664 - acc: 0.9278 - val_loss: 0.2360 - val_acc: 0.9460
    Epoch 9/25
    57000/57000 [==============================] - 1s - loss: 0.2631 - acc: 0.9295 - val_loss: 0.2377 - val_acc: 0.9410
    Epoch 10/25
    57000/57000 [==============================] - 1s - loss: 0.2615 - acc: 0.9296 - val_loss: 0.2355 - val_acc: 0.9437
    Epoch 11/25
    57000/57000 [==============================] - 1s - loss: 0.2597 - acc: 0.9300 - val_loss: 0.2457 - val_acc: 0.9430
    Epoch 12/25
    57000/57000 [==============================] - 1s - loss: 0.2585 - acc: 0.9306 - val_loss: 0.2405 - val_acc: 0.9447
    Epoch 13/25
    57000/57000 [==============================] - 1s - loss: 0.2570 - acc: 0.9306 - val_loss: 0.2445 - val_acc: 0.9390
    Epoch 14/25
    57000/57000 [==============================] - 1s - loss: 0.2554 - acc: 0.9315 - val_loss: 0.2376 - val_acc: 0.9453
    Epoch 15/25
    57000/57000 [==============================] - 1s - loss: 0.2544 - acc: 0.9308 - val_loss: 0.2465 - val_acc: 0.9440
    Epoch 16/25
    57000/57000 [==============================] - 1s - loss: 0.2534 - acc: 0.9323 - val_loss: 0.2393 - val_acc: 0.9410
    Epoch 17/25
    57000/57000 [==============================] - 1s - loss: 0.2527 - acc: 0.9317 - val_loss: 0.2495 - val_acc: 0.9393
    Epoch 18/25
    57000/57000 [==============================] - 1s - loss: 0.2524 - acc: 0.9324 - val_loss: 0.2471 - val_acc: 0.9403
    Epoch 19/25
    57000/57000 [==============================] - 1s - loss: 0.2512 - acc: 0.9321 - val_loss: 0.2505 - val_acc: 0.9407
    Epoch 20/25
    57000/57000 [==============================] - 1s - loss: 0.2505 - acc: 0.9318 - val_loss: 0.2475 - val_acc: 0.9413
    Epoch 21/25
    57000/57000 [==============================] - 1s - loss: 0.2497 - acc: 0.9334 - val_loss: 0.2487 - val_acc: 0.9433
    Epoch 22/25
    57000/57000 [==============================] - 1s - loss: 0.2489 - acc: 0.9325 - val_loss: 0.2483 - val_acc: 0.9433
    Epoch 23/25
    57000/57000 [==============================] - 1s - loss: 0.2487 - acc: 0.9328 - val_loss: 0.2518 - val_acc: 0.9420
    Epoch 24/25
    57000/57000 [==============================] - 1s - loss: 0.2482 - acc: 0.9325 - val_loss: 0.2574 - val_acc: 0.9400
    Epoch 25/25
    57000/57000 [==============================] - 1s - loss: 0.2483 - acc: 0.9329 - val_loss: 0.2508 - val_acc: 0.9433



```python
plt.plot(history.history['acc'])
```


```python
plt.plot(history.history['val_acc'])
```


```python
deep = Sequential()
```


```python
deep.add(Dense(256, activation = 'relu', input_dim = (28**2)))
```


```python
deep.add(Dropout(0.2))
```


```python
deep.add(Dense(128,activation = 'relu'))
```


```python
deep.add(Dropout(0.2))
```


```python
deep.add(Dense(64,activation = 'relu'))
```


```python
deep.add(Dropout(0.2))
```


```python
deep.add(Dense(10,activation = 'softmax'))
```


```python
deep.compile(optimizer=Adam(),
             loss='categorical_crossentropy',
             metrics=['accuracy'])
```


```python
history2=deep.fit(train_x_tr, y_train_ohe, validation_split=0.05,
                 epochs=25, batch_size=64)
```

    Train on 57000 samples, validate on 3000 samples
    Epoch 1/25
    57000/57000 [==============================] - 4s - loss: 0.3828 - acc: 0.8863 - val_loss: 0.1097 - val_acc: 0.9683
    Epoch 2/25
    57000/57000 [==============================] - 4s - loss: 0.1952 - acc: 0.9437 - val_loss: 0.0968 - val_acc: 0.9753
    Epoch 3/25
    57000/57000 [==============================] - 4s - loss: 0.1520 - acc: 0.9555 - val_loss: 0.0861 - val_acc: 0.9757
    Epoch 4/25
    57000/57000 [==============================] - 4s - loss: 0.1281 - acc: 0.9632 - val_loss: 0.0875 - val_acc: 0.9773
    Epoch 5/25
    57000/57000 [==============================] - 4s - loss: 0.1121 - acc: 0.9671 - val_loss: 0.0897 - val_acc: 0.9777
    Epoch 6/25
    57000/57000 [==============================] - 4s - loss: 0.0995 - acc: 0.9714 - val_loss: 0.0950 - val_acc: 0.9783
    Epoch 7/25
    57000/57000 [==============================] - 4s - loss: 0.0922 - acc: 0.9731 - val_loss: 0.0810 - val_acc: 0.9800
    Epoch 8/25
    57000/57000 [==============================] - 4s - loss: 0.0840 - acc: 0.9749 - val_loss: 0.0852 - val_acc: 0.9820
    Epoch 9/25
    57000/57000 [==============================] - 4s - loss: 0.0765 - acc: 0.9774 - val_loss: 0.0766 - val_acc: 0.9840
    Epoch 10/25
    57000/57000 [==============================] - 4s - loss: 0.0769 - acc: 0.9783 - val_loss: 0.0831 - val_acc: 0.9810
    Epoch 11/25
    57000/57000 [==============================] - 4s - loss: 0.0690 - acc: 0.9798 - val_loss: 0.0845 - val_acc: 0.9810
    Epoch 12/25
    57000/57000 [==============================] - 4s - loss: 0.0604 - acc: 0.9822 - val_loss: 0.0982 - val_acc: 0.9817
    Epoch 13/25
    57000/57000 [==============================] - 4s - loss: 0.0595 - acc: 0.9822 - val_loss: 0.0977 - val_acc: 0.9807
    Epoch 14/25
    57000/57000 [==============================] - 4s - loss: 0.0553 - acc: 0.9835 - val_loss: 0.0810 - val_acc: 0.9823
    Epoch 15/25
    57000/57000 [==============================] - 4s - loss: 0.0551 - acc: 0.9845 - val_loss: 0.0899 - val_acc: 0.9807
    Epoch 16/25
    57000/57000 [==============================] - 3s - loss: 0.0532 - acc: 0.9840 - val_loss: 0.0794 - val_acc: 0.9817
    Epoch 17/25
    57000/57000 [==============================] - 3s - loss: 0.0500 - acc: 0.9856 - val_loss: 0.0840 - val_acc: 0.9787
    Epoch 18/25
    57000/57000 [==============================] - 3s - loss: 0.0480 - acc: 0.9860 - val_loss: 0.0849 - val_acc: 0.9827
    Epoch 19/25
    57000/57000 [==============================] - 3s - loss: 0.0456 - acc: 0.9863 - val_loss: 0.0875 - val_acc: 0.9817
    Epoch 20/25
    57000/57000 [==============================] - 3s - loss: 0.0442 - acc: 0.9873 - val_loss: 0.1048 - val_acc: 0.9803
    Epoch 21/25
    57000/57000 [==============================] - 3s - loss: 0.0446 - acc: 0.9875 - val_loss: 0.0860 - val_acc: 0.9837
    Epoch 22/25
    57000/57000 [==============================] - 3s - loss: 0.0422 - acc: 0.9878 - val_loss: 0.0890 - val_acc: 0.9830
    Epoch 23/25
    57000/57000 [==============================] - 3s - loss: 0.0409 - acc: 0.9877 - val_loss: 0.0892 - val_acc: 0.9823
    Epoch 24/25
    57000/57000 [==============================] - 4s - loss: 0.0394 - acc: 0.9880 - val_loss: 0.0957 - val_acc: 0.9843
    Epoch 25/25
    57000/57000 [==============================] - 4s - loss: 0.0381 - acc: 0.9890 - val_loss: 0.0872 - val_acc: 0.9837



```python
from keras.layers import Conv2D, GlobalAveragePooling2D
```


```python
train_x_re = train_x_tr.reshape(-1,28,28,1)
```


```python
CNN = Sequential()
```


```python
CNN.add(Conv2D(filters = 16, kernel_size = (3,3), padding = 'SAME',input_shape = (28,28,1), activation = 'relu'))
```


```python
CNN.add(Conv2D(filters = 32, kernel_size = (3,3), padding = 'SAME', activation = 'relu'))
```


```python
CNN.add(Conv2D(filters = 64, kernel_size = (3,3), padding = 'SAME', activation = 'relu'))
```


```python
CNN.add(GlobalAveragePooling2D())
```


```python
CNN.add(Dense(10,activation='softmax'))
```


```python
CNN.compile(optimizer=Adam(),
             loss='categorical_crossentropy',
             metrics=['accuracy'])
```


```python
history3=CNN.fit(train_x_re, y_train_ohe, validation_split=0.05,
                 epochs=25, batch_size=64)
```

    Train on 57000 samples, validate on 3000 samples
    Epoch 1/25
    57000/57000 [==============================] - 88s - loss: 1.4696 - acc: 0.4872 - val_loss: 0.8768 - val_acc: 0.7490
    Epoch 2/25
    57000/57000 [==============================] - 86s - loss: 0.8037 - acc: 0.7588 - val_loss: 0.5660 - val_acc: 0.8610
    Epoch 3/25
    57000/57000 [==============================] - 86s - loss: 0.5973 - acc: 0.8276 - val_loss: 0.5143 - val_acc: 0.8413
    Epoch 4/25
    57000/57000 [==============================] - 86s - loss: 0.4874 - acc: 0.8607 - val_loss: 0.3775 - val_acc: 0.8937
    Epoch 5/25
    57000/57000 [==============================] - 86s - loss: 0.4188 - acc: 0.8808 - val_loss: 0.3267 - val_acc: 0.9067
    Epoch 6/25
    57000/57000 [==============================] - 86s - loss: 0.3590 - acc: 0.8967 - val_loss: 0.3364 - val_acc: 0.8970
    Epoch 7/25
    57000/57000 [==============================] - 86s - loss: 0.3227 - acc: 0.9066 - val_loss: 0.2335 - val_acc: 0.9323
    Epoch 8/25
    57000/57000 [==============================] - 86s - loss: 0.2887 - acc: 0.9172 - val_loss: 0.2236 - val_acc: 0.9377
    Epoch 9/25
    57000/57000 [==============================] - 87s - loss: 0.2629 - acc: 0.9255 - val_loss: 0.2114 - val_acc: 0.9413
    Epoch 10/25
    57000/57000 [==============================] - 85s - loss: 0.2467 - acc: 0.9295 - val_loss: 0.1958 - val_acc: 0.9460
    Epoch 11/25
    57000/57000 [==============================] - 84s - loss: 0.2231 - acc: 0.9365 - val_loss: 0.1812 - val_acc: 0.9527
    Epoch 12/25
    57000/57000 [==============================] - 84s - loss: 0.2068 - acc: 0.9406 - val_loss: 0.1624 - val_acc: 0.9553
    Epoch 13/25
    57000/57000 [==============================] - 83s - loss: 0.1974 - acc: 0.9442 - val_loss: 0.1693 - val_acc: 0.9533
    Epoch 14/25
    57000/57000 [==============================] - 83s - loss: 0.1861 - acc: 0.9469 - val_loss: 0.1706 - val_acc: 0.9530
    Epoch 15/25
    57000/57000 [==============================] - 83s - loss: 0.1769 - acc: 0.9495 - val_loss: 0.1247 - val_acc: 0.9667
    Epoch 16/25
    57000/57000 [==============================] - 84s - loss: 0.1663 - acc: 0.9520 - val_loss: 0.1521 - val_acc: 0.9610
    Epoch 17/25
    57000/57000 [==============================] - 84s - loss: 0.1575 - acc: 0.9546 - val_loss: 0.1330 - val_acc: 0.9627
    Epoch 18/25
    57000/57000 [==============================] - 84s - loss: 0.1514 - acc: 0.9560 - val_loss: 0.1134 - val_acc: 0.9690
    Epoch 19/25
    57000/57000 [==============================] - 84s - loss: 0.1432 - acc: 0.9586 - val_loss: 0.1221 - val_acc: 0.9677
    Epoch 20/25
    57000/57000 [==============================] - 84s - loss: 0.1390 - acc: 0.9590 - val_loss: 0.1339 - val_acc: 0.9640
    Epoch 21/25
    57000/57000 [==============================] - 84s - loss: 0.1314 - acc: 0.9613 - val_loss: 0.1105 - val_acc: 0.9683
    Epoch 22/25
    57000/57000 [==============================] - 84s - loss: 0.1270 - acc: 0.9624 - val_loss: 0.0966 - val_acc: 0.9740
    Epoch 23/25
    57000/57000 [==============================] - 84s - loss: 0.1258 - acc: 0.9636 - val_loss: 0.1046 - val_acc: 0.9677
    Epoch 24/25
    57000/57000 [==============================] - 84s - loss: 0.1161 - acc: 0.9663 - val_loss: 0.0900 - val_acc: 0.9757
    Epoch 25/25
    57000/57000 [==============================] - 84s - loss: 0.1148 - acc: 0.9663 - val_loss: 0.0962 - val_acc: 0.9747


726ded21ae79fe04fe917ed86ebd4b84dfe03acd
ipynb
Jupyter Notebook
exercise-sessions/Session-13/Session-13.ipynb
JustinNoel1/ML-Course
df805fea8febe6a92ca08142b93460432f6aaed0
Apache-2.0
exercise-sessions/Session-13/Session-13.ipynb
JustinNoel1/ML-Course
df805fea8febe6a92ca08142b93460432f6aaed0
Apache-2.0
exercise-sessions/Session-13/Session-13.ipynb
JustinNoel1/ML-Course
df805fea8febe6a92ca08142b93460432f6aaed0
Apache-2.0
Qwen/Qwen-72B
1. YES
2. YES