# UName : Mahesh Gosi
# UBit No : 50290934
# UBit Name: mgosi

# Introduction to Pattern Recognition
## Assignment - 4

### Task 1: 

Neural networks that have one hidden layer are generally known for universal approximation, they can approximate any continuos function. This approximation can be improved by increasing the number of hidden neurons on the network. There is a risk involved with this kind of process, that is a risk of overfitting.

A key feature of neural network is that they are able to learn features independently without much human involvement. 

__Softmax function:__
- The softmax function is used as the output layer's activation function. This function will calculate the probabilities of each target class over all possible target classes. This helps us determine the target class for the inputs.
- The sum of the probabilities is always equal to 1, as it is a list of probabilities for each class in the system.



__Negative Log-likelihood:__

- We find the log likelihood and maximize the value to find the parameters of the data. We can also do this by taking the negative log likelihood and minimizing this value.  
- This is done by the softmax function to find the maximum likelihood given the data. 

We update the weights of the neural network through backpropagation. We find the difference between the new weights and current weights and calculate the loss. We propagate this loss to update the weights of the network. Hence by minimizing the loss, we obtain a more accurate model while training.

Thus the negative log likelihood helps us get the loss of the neural network when we pass it through the softmax output nodes, by giving the confidence of the likely class and then computing the log of the class.

__When given a guassian prior of weight distribution:__

Consider that we want to infer some parameter $\beta $ from some observed input-output pairs. Let us assume that the outputs are linearly related to inputs with some noise $\epsilon $:

\begin{align}
 y_n = \beta x_n + \epsilon 
\end{align}

where $\epsilon$ is the guassian noise. From this we can compute the gussian likelihood:

\begin{align}
 \prod_{n=1}^{N} \mathcal{N}(y_n | \beta x_n, \sigma^2 )
\end{align}


Now our next step is regularize the parameter $\beta$ by imposing a guassian prior $\mathcal{N}(\beta|0, \lambda^-1) $
, where $\lambda $ is strictly positive scalar. Hence by combining the likelihood and prior, we have

\begin{align}
 \prod_{n=1}^{N} \mathcal{N}(y_n | \beta x_n, \sigma^2 ) \mathcal{N}(\beta|0, \lambda^-1) 
\end{align}

Now, taking the log of the above expression and ignoring some constants, we have:


\begin{align}
\sum_{n=1}^N -\dfrac{1}{\sigma^2}(y_n - \beta x_n)^2 -\lambda \beta^2 + c
\end{align}

If we maximize the above expression with respect to $\beta$, we get __Maximum a-posteriori estimate (MAP)__. From this we can say that the gaussian prior can be interpreted as a L2 regularization term.

### Task 2


```python
import tensorflow as tf
import numpy as np
import pandas as pd
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.utils import np_utils
from keras.optimizers import SGD
import matplotlib.pyplot as plt
import keras
import keras.backend as K
from keras.callbacks import LambdaCallback
from keras.regularizers import l2
import plotly
import plotly.plotly as py
import plotly.graph_objs as go
```

    Using TensorFlow backend.



```python
(x_train, y_train), (x_test, y_test) = mnist.load_data()
```

#### Making test sets of 100 images each


```python
XTrain = x_train[y_train[:,]==0][:100]
YTrain = np.zeros(100).tolist()
XTest = x_test[y_test[:,]==0][:100]
YTest = np.zeros(100).tolist()
YTemp = np.empty(100)
for i in range(1,10):
    XTrain = np.concatenate((XTrain, x_train[y_train[:,]==i][:100]))
    YTemp.fill(i)
    YTrain = YTrain + YTemp.tolist()
    XTest = np.concatenate((XTest, x_test[y_test[:,]==i][:100] ))
    YTest = YTest + YTemp.tolist()

XTrain = XTrain.reshape(XTrain.shape[0], XTrain.shape[1]*XTrain.shape[2]).astype('float32')/255
XTest = XTest.reshape(XTest.shape[0], XTest.shape[1]*XTest.shape[2]).astype('float32')/255
YTrain = np_utils.to_categorical(YTrain)
YTest = np_utils.to_categorical(YTest)
XTrain.shape, XTest.shape, YTrain.shape, YTest.shape
```




    ((1000, 784), (1000, 784), (1000, 10), (1000, 10))



#### Taking 1000 labels


```python
numClasses=YTest.shape[1]
noOfPixels = 784
```


```python
learn_rate=[]
# Class to generate the learining rate based on the callback. 
class SGDLearningRateTracker(keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs={}):
        optimizer=self.model.optimizer
        _lr=tf.to_float(optimizer.lr, name='ToFloat')
        _decay=tf.to_float(optimizer.decay, name='ToFloat')
        _iter=tf.to_float(optimizer.iterations, name='ToFloat')
        
        lr=K.eval(_lr *(1./(1. + _decay*_iter)))
        learn_rate.append(lr)
```


```python
loss_test=[]
acc_test=[]

# Classes to generate the loss for training and testing.
class TestCallback_test(keras.callbacks.Callback):
    def __init__(self, test_data):
        self.test_data=test_data
        
    def on_epoch_end(self, epoch, logs={}):
        x,y = self.test_data
        loss, acc= self.model.evaluate(x,y, verbose=0)
        loss_test.append(loss)
        acc_test.append(acc)
        
loss_train=[]
acc_train=[]
class TestCallback_train(keras.callbacks.Callback):
    def __init__(self, test_data):
        self.test_data=test_data
        
    def on_epoch_end(self, epoch, logs={}):
        x,y = self.test_data
        loss, acc= self.model.evaluate(x,y, verbose=0)
        loss_train.append(loss)
        acc_train.append(acc)
```


```python
# ==== ALL MODELS ===#
def Task2_a():
    model = Sequential()
    model.add(Dense(30, input_dim=noOfPixels, kernel_initializer='normal', activation='sigmoid'))
    model.add(Dense(numClasses, kernel_initializer='normal', activation='softmax'))
    model.summary()
    model.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.1,decay=0.001), metrics=['accuracy'])
    return model

def Task2_b1():
    model = Sequential()
    model.add(Dense(30, input_dim=noOfPixels, kernel_initializer='normal', activation='sigmoid'))
    model.add(Dense(30, kernel_initializer='normal', activation='sigmoid'))
    model.add(Dense(numClasses, kernel_initializer='normal', activation='softmax'))
    model.summary()
    model.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.1,decay=0.00001), metrics=['accuracy'])
    return model

def Task2_b2():
    model = Sequential()
    model.add(Dense(30, input_dim=noOfPixels, kernel_initializer='normal', activation='sigmoid'))
    model.add(Dense(30, kernel_initializer='normal', activation='sigmoid'))
    model.add(Dense(30, kernel_initializer='normal', activation='sigmoid'))
    model.add(Dense(numClasses, kernel_initializer='normal', activation='softmax'))
    model.summary()
    model.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.1,decay=0.1), metrics=['accuracy'])
    return model

def Task2_b3():
    model = Sequential()
    model.add(Dense(30, input_dim=noOfPixels, kernel_initializer='normal', activation='sigmoid',W_regularizer=l2(5)))
    model.add(Dense(numClasses, kernel_initializer='normal', activation='softmax'))
    model.summary()
    model.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.1,decay=0.001), metrics=['accuracy'])
    return model

def Task2_b4():
    model = Sequential()
    model.add(Dense(30, input_dim=noOfPixels, kernel_initializer='normal', activation='sigmoid',W_regularizer=l2(5)))
    model.add(Dense(30, kernel_initializer='normal', activation='sigmoid',W_regularizer=l2(5)))
    model.add(Dense(numClasses, kernel_initializer='normal', activation='softmax'))
    model.summary()
    model.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.1,decay=0.00001), metrics=['accuracy'])
    return model

def Task2_b5():
    model = Sequential()
    model.add(Dense(30, input_dim=noOfPixels, kernel_initializer='normal', activation='sigmoid',W_regularizer=l2(5)))
    model.add(Dense(30, kernel_initializer='normal', activation='sigmoid',W_regularizer=l2(5)))
    model.add(Dense(30, kernel_initializer='normal', activation='sigmoid',W_regularizer=l2(5)))
    model.add(Dense(numClasses, kernel_initializer='normal', activation='softmax'))
    model.summary()
    model.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.1,decay=0.0001), metrics=['accuracy'])
    return model
```


```python
callbacks_list1= [SGDLearningRateTracker(), TestCallback_test((XTest, YTest)), TestCallback_train((XTrain, YTrain))]
callbacks_list2= [SGDLearningRateTracker(), TestCallback_test((XTest, YTest)), TestCallback_train((XTrain, YTrain))]
```


```python
epoch=[]
for i in range(1,31):
    epoch.append(i)
  
```


```python
#========== Task 2(a) ===============#
model1=Task2_a()
history1=model1.fit(XTrain, YTrain, validation_data=(XTest, YTest), nb_epoch=30, batch_size=10,
verbose=1, callbacks=callbacks_list1)
```

    WARNING:tensorflow:From /Users/akash/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Colocations handled automatically by placer.
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #   
    =================================================================
    dense_1 (Dense)              (None, 30)                23550     
    _________________________________________________________________
    dense_2 (Dense)              (None, 10)                310       
    =================================================================
    Total params: 23,860
    Trainable params: 23,860
    Non-trainable params: 0
    _________________________________________________________________
    WARNING:tensorflow:From /Users/akash/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.cast instead.


    /Users/akash/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:4: UserWarning:
    
    The `nb_epoch` argument in `fit` has been renamed `epochs`.
    


    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
    1000/1000 [==============================] - 2s 2ms/step - loss: 2.2783 - acc: 0.1240 - val_loss: 2.2121 - val_acc: 0.1750
    WARNING:tensorflow:From <ipython-input-5-e484a104638f>:6: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.cast instead.
    Epoch 2/30
    1000/1000 [==============================] - 1s 996us/step - loss: 2.0619 - acc: 0.4080 - val_loss: 1.9161 - val_acc: 0.5590
    Epoch 3/30
    1000/1000 [==============================] - 0s 477us/step - loss: 1.6692 - acc: 0.6340 - val_loss: 1.5476 - val_acc: 0.6330
    Epoch 4/30
    1000/1000 [==============================] - 1s 589us/step - loss: 1.3157 - acc: 0.6960 - val_loss: 1.3011 - val_acc: 0.6810
    Epoch 5/30
    1000/1000 [==============================] - 1s 829us/step - loss: 1.0782 - acc: 0.7680 - val_loss: 1.1187 - val_acc: 0.7040
    Epoch 6/30
    1000/1000 [==============================] - 1s 665us/step - loss: 0.9144 - acc: 0.8040 - val_loss: 0.9981 - val_acc: 0.7220
    Epoch 7/30
    1000/1000 [==============================] - 1s 664us/step - loss: 0.7997 - acc: 0.8410 - val_loss: 0.9203 - val_acc: 0.7450
    Epoch 8/30
    1000/1000 [==============================] - 1s 524us/step - loss: 0.7121 - acc: 0.8500 - val_loss: 0.8591 - val_acc: 0.7600
    Epoch 9/30
    1000/1000 [==============================] - 1s 522us/step - loss: 0.6481 - acc: 0.8610 - val_loss: 0.8077 - val_acc: 0.7680
    Epoch 10/30
    1000/1000 [==============================] - 1s 616us/step - loss: 0.5982 - acc: 0.8670 - val_loss: 0.7688 - val_acc: 0.7830
    Epoch 11/30
    1000/1000 [==============================] - 0s 486us/step - loss: 0.5556 - acc: 0.8770 - val_loss: 0.7378 - val_acc: 0.7910
    Epoch 12/30
    1000/1000 [==============================] - 0s 494us/step - loss: 0.5211 - acc: 0.8810 - val_loss: 0.7114 - val_acc: 0.7880
    Epoch 13/30
    1000/1000 [==============================] - 1s 632us/step - loss: 0.4928 - acc: 0.8880 - val_loss: 0.6899 - val_acc: 0.8010
    Epoch 14/30
    1000/1000 [==============================] - 1s 535us/step - loss: 0.4665 - acc: 0.8890 - val_loss: 0.6696 - val_acc: 0.8060
    Epoch 15/30
    1000/1000 [==============================] - 1s 557us/step - loss: 0.4462 - acc: 0.8930 - val_loss: 0.6536 - val_acc: 0.8100
    Epoch 16/30
    1000/1000 [==============================] - 1s 609us/step - loss: 0.4265 - acc: 0.9020 - val_loss: 0.6386 - val_acc: 0.8060
    Epoch 17/30
    1000/1000 [==============================] - 0s 479us/step - loss: 0.4095 - acc: 0.9050 - val_loss: 0.6297 - val_acc: 0.8100
    Epoch 18/30
    1000/1000 [==============================] - 0s 487us/step - loss: 0.3949 - acc: 0.9140 - val_loss: 0.6185 - val_acc: 0.8150
    Epoch 19/30
    1000/1000 [==============================] - 1s 540us/step - loss: 0.3819 - acc: 0.9170 - val_loss: 0.6072 - val_acc: 0.8170
    Epoch 20/30
    1000/1000 [==============================] - 0s 490us/step - loss: 0.3703 - acc: 0.9160 - val_loss: 0.5990 - val_acc: 0.8230
    Epoch 21/30
    1000/1000 [==============================] - 0s 474us/step - loss: 0.3579 - acc: 0.9240 - val_loss: 0.5936 - val_acc: 0.8210
    Epoch 22/30
    1000/1000 [==============================] - 1s 575us/step - loss: 0.3490 - acc: 0.9250 - val_loss: 0.5859 - val_acc: 0.8250
    Epoch 23/30
    1000/1000 [==============================] - 0s 484us/step - loss: 0.3388 - acc: 0.9270 - val_loss: 0.5785 - val_acc: 0.8230
    Epoch 24/30
    1000/1000 [==============================] - 1s 602us/step - loss: 0.3310 - acc: 0.9290 - val_loss: 0.5728 - val_acc: 0.8230
    Epoch 25/30
    1000/1000 [==============================] - 1s 537us/step - loss: 0.3225 - acc: 0.9310 - val_loss: 0.5661 - val_acc: 0.8300
    Epoch 26/30
    1000/1000 [==============================] - 1s 538us/step - loss: 0.3157 - acc: 0.9330 - val_loss: 0.5634 - val_acc: 0.8300
    Epoch 27/30
    1000/1000 [==============================] - 1s 697us/step - loss: 0.3085 - acc: 0.9310 - val_loss: 0.5575 - val_acc: 0.8330
    Epoch 28/30
    1000/1000 [==============================] - 0s 497us/step - loss: 0.3019 - acc: 0.9350 - val_loss: 0.5552 - val_acc: 0.8330
    Epoch 29/30
    1000/1000 [==============================] - 0s 497us/step - loss: 0.2961 - acc: 0.9380 - val_loss: 0.5496 - val_acc: 0.8320
    Epoch 30/30
    1000/1000 [==============================] - 1s 643us/step - loss: 0.2903 - acc: 0.9380 - val_loss: 0.5466 - val_acc: 0.8340



```python
plotly.tools.set_credentials_file(username='mgosi', api_key='X3dKXRZPub6oHfIoFii7')
trace1 = go.Scatter(
    x=epoch,
    y=learn_rate,
    name = '<b>2a Learning Rate</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace2 = go.Scatter(
    x=epoch,
    y=loss_train,
    name = '<b>2a Training Loss</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace3 = go.Scatter(
    x=epoch,
    y=loss_test,
    name = '<b>2a Testing Loss</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace4 = go.Scatter(
    x=epoch,
    y=history1.history["val_acc"],
    name = '<b>2a Testing Accuracy</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace5 = go.Scatter(
    x=epoch,
    y=history1.history["acc"],
    name = '<b>2a Training Accuracy</b>', # Style name/legend entry with html tags
    connectgaps=True
)
data = [trace1, trace2, trace3, trace4, trace5]
layout = dict(title = 'Task 2a',
              xaxis = dict(title = 'Epoch'),
              yaxis = dict(title = 'Loss'),
              )
fig = dict(data=data, layout = layout)
py.iplot(fig, filename='Task2a')
```

    /Users/akash/anaconda3/lib/python3.7/site-packages/IPython/core/display.py:689: UserWarning:
    
    Consider using IPython.display.IFrame instead
    










```python
plotly.tools.set_credentials_file(username='mgosi', api_key='X3dKXRZPub6oHfIoFii7')
trace1 = go.Scatter(
    x=epoch,
    y=learn_rate,
    name = '<b>2a Learning Rate</b>', # Style name/legend entry with html tags
    connectgaps=True
)
data = [trace1]
layout = dict(title = 'Task 2a Learning Rate',
              xaxis = dict(title = 'Epoch'),
              yaxis = dict(title = 'Rate'),
              )
fig = dict(data=data, layout = layout)
py.iplot(fig, filename='Task 2a Learning Rate')
```








## Two Hidden layers


```python
loss_test=[]
acc_test=[]
learn_rate=[]
loss_train=[]
acc_train=[]

model2=Task2_b1()
history2=model2.fit(XTrain, YTrain, validation_data=(XTest, YTest), nb_epoch=30, batch_size=10,
verbose=2, callbacks=callbacks_list2)
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #   
    =================================================================
    dense_3 (Dense)              (None, 30)                23550     
    _________________________________________________________________
    dense_4 (Dense)              (None, 30)                930       
    _________________________________________________________________
    dense_5 (Dense)              (None, 10)                310       
    =================================================================
    Total params: 24,790
    Trainable params: 24,790
    Non-trainable params: 0
    _________________________________________________________________


    /Users/akash/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:9: UserWarning:
    
    The `nb_epoch` argument in `fit` has been renamed `epochs`.
    


    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
     - 2s - loss: 2.3260 - acc: 0.0850 - val_loss: 2.3149 - val_acc: 0.1000
    Epoch 2/30
     - 0s - loss: 2.3253 - acc: 0.0890 - val_loss: 2.3133 - val_acc: 0.1000
    Epoch 3/30
     - 0s - loss: 2.3232 - acc: 0.0920 - val_loss: 2.3115 - val_acc: 0.1000
    Epoch 4/30
     - 1s - loss: 2.3173 - acc: 0.0970 - val_loss: 2.3221 - val_acc: 0.1000
    Epoch 5/30
     - 0s - loss: 2.3171 - acc: 0.0950 - val_loss: 2.3140 - val_acc: 0.1000
    Epoch 6/30
     - 0s - loss: 2.3161 - acc: 0.0970 - val_loss: 2.3230 - val_acc: 0.1000
    Epoch 7/30
     - 1s - loss: 2.3132 - acc: 0.0940 - val_loss: 2.2978 - val_acc: 0.1870
    Epoch 8/30
     - 0s - loss: 2.3103 - acc: 0.0910 - val_loss: 2.2934 - val_acc: 0.1000
    Epoch 9/30
     - 0s - loss: 2.2948 - acc: 0.1060 - val_loss: 2.2900 - val_acc: 0.1280
    Epoch 10/30
     - 1s - loss: 2.2756 - acc: 0.1500 - val_loss: 2.2610 - val_acc: 0.2100
    Epoch 11/30
     - 0s - loss: 2.2475 - acc: 0.1930 - val_loss: 2.2250 - val_acc: 0.2620
    Epoch 12/30
     - 0s - loss: 2.1775 - acc: 0.2700 - val_loss: 2.1286 - val_acc: 0.3860
    Epoch 13/30
     - 1s - loss: 2.0238 - acc: 0.3810 - val_loss: 1.9524 - val_acc: 0.4170
    Epoch 14/30
     - 0s - loss: 1.8053 - acc: 0.4650 - val_loss: 1.7371 - val_acc: 0.4260
    Epoch 15/30
     - 0s - loss: 1.5683 - acc: 0.5380 - val_loss: 1.5413 - val_acc: 0.4960
    Epoch 16/30
     - 0s - loss: 1.3737 - acc: 0.5830 - val_loss: 1.3781 - val_acc: 0.5570
    Epoch 17/30
     - 0s - loss: 1.2199 - acc: 0.6240 - val_loss: 1.2610 - val_acc: 0.5820
    Epoch 18/30
     - 0s - loss: 1.0994 - acc: 0.6500 - val_loss: 1.1779 - val_acc: 0.5690
    Epoch 19/30
     - 0s - loss: 1.0118 - acc: 0.6720 - val_loss: 1.1202 - val_acc: 0.6080
    Epoch 20/30
     - 0s - loss: 0.9373 - acc: 0.7120 - val_loss: 1.0788 - val_acc: 0.5950
    Epoch 21/30
     - 0s - loss: 0.8821 - acc: 0.7220 - val_loss: 1.0415 - val_acc: 0.6400
    Epoch 22/30
     - 1s - loss: 0.8324 - acc: 0.7280 - val_loss: 1.0188 - val_acc: 0.6340
    Epoch 23/30
     - 1s - loss: 0.7861 - acc: 0.7590 - val_loss: 1.0041 - val_acc: 0.6360
    Epoch 24/30
     - 1s - loss: 0.7513 - acc: 0.7710 - val_loss: 0.9692 - val_acc: 0.6700
    Epoch 25/30
     - 1s - loss: 0.7078 - acc: 0.7900 - val_loss: 0.9714 - val_acc: 0.6720
    Epoch 26/30
     - 0s - loss: 0.6763 - acc: 0.8120 - val_loss: 0.9594 - val_acc: 0.6840
    Epoch 27/30
     - 0s - loss: 0.6427 - acc: 0.8240 - val_loss: 0.9084 - val_acc: 0.6910
    Epoch 28/30
     - 1s - loss: 0.6101 - acc: 0.8300 - val_loss: 0.9054 - val_acc: 0.6980
    Epoch 29/30
     - 0s - loss: 0.5771 - acc: 0.8470 - val_loss: 0.8700 - val_acc: 0.7270
    Epoch 30/30
     - 0s - loss: 0.5432 - acc: 0.8670 - val_loss: 0.8551 - val_acc: 0.7370



```python
plotly.tools.set_credentials_file(username='mgosi', api_key='X3dKXRZPub6oHfIoFii7')
trace1 = go.Scatter(
    x=epoch,
    y=learn_rate,
    name = '<b>2b1 Learning Rate</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace2 = go.Scatter(
    x=epoch,
    y=loss_train,
    name = '<b>2b1 Training Loss</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace3 = go.Scatter(
    x=epoch,
    y=loss_test,
    name = '<b>2b1 Testing Loss</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace4 = go.Scatter(
    x=epoch,
    y=history2.history["val_acc"],
    name = '<b>2b1 Testing Accuracy</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace5 = go.Scatter(
    x=epoch,
    y=history2.history["acc"],
    name = '<b>2b1 Training Accuracy</b>', # Style name/legend entry with html tags
    connectgaps=True
)
data = [trace1, trace2, trace3, trace4, trace5]
layout = dict(title = 'Task 2b1',
              xaxis = dict(title = 'Epoch'),
              yaxis = dict(title = 'Loss'),
              )
fig = dict(data=data, layout = layout)
py.iplot(fig, filename='Task 2b1')
```









```python
plotly.tools.set_credentials_file(username='mgosi', api_key='X3dKXRZPub6oHfIoFii7')
trace1 = go.Scatter(
    x=epoch,
    y=learn_rate,
    name = '<b>2b1 Learning Rate</b>', # Style name/legend entry with html tags
    connectgaps=True
)
data = [trace1]
layout = dict(title = 'Task 2b1 Learning Rate',
              xaxis = dict(title = 'Epoch'),
              yaxis = dict(title = 'Rate'),
              )
fig = dict(data=data, layout = layout)
py.iplot(fig, filename='Task 2b1 Learning Rate')
```








## Three Hidden Layers


```python
loss_test=[]
acc_test=[]
learn_rate=[]
loss_train=[]
acc_train=[]

model3=Task2_b2()
history3=model3.fit(XTrain, YTrain, validation_data=(XTest, YTest), nb_epoch=30, batch_size=10,
verbose=2, callbacks=callbacks_list1)
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #   
    =================================================================
    dense_6 (Dense)              (None, 30)                23550     
    _________________________________________________________________
    dense_7 (Dense)              (None, 30)                930       
    _________________________________________________________________
    dense_8 (Dense)              (None, 30)                930       
    _________________________________________________________________
    dense_9 (Dense)              (None, 10)                310       
    =================================================================
    Total params: 25,720
    Trainable params: 25,720
    Non-trainable params: 0
    _________________________________________________________________


    /Users/akash/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:9: UserWarning:
    
    The `nb_epoch` argument in `fit` has been renamed `epochs`.
    


    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
     - 2s - loss: 2.3130 - acc: 0.0880 - val_loss: 2.3027 - val_acc: 0.1000
    Epoch 2/30
     - 0s - loss: 2.3052 - acc: 0.0780 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 3/30
     - 1s - loss: 2.3042 - acc: 0.0920 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 4/30
     - 0s - loss: 2.3037 - acc: 0.0890 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 5/30
     - 0s - loss: 2.3035 - acc: 0.0800 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 6/30
     - 1s - loss: 2.3033 - acc: 0.0790 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 7/30
     - 0s - loss: 2.3032 - acc: 0.0770 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 8/30
     - 0s - loss: 2.3031 - acc: 0.0790 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 9/30
     - 1s - loss: 2.3030 - acc: 0.0820 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 10/30
     - 0s - loss: 2.3030 - acc: 0.0870 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 11/30
     - 0s - loss: 2.3029 - acc: 0.0780 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 12/30
     - 1s - loss: 2.3029 - acc: 0.0920 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 13/30
     - 0s - loss: 2.3029 - acc: 0.0790 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 14/30
     - 1s - loss: 2.3029 - acc: 0.0880 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 15/30
     - 1s - loss: 2.3028 - acc: 0.0820 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 16/30
     - 1s - loss: 2.3028 - acc: 0.0700 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 17/30
     - 1s - loss: 2.3028 - acc: 0.0930 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 18/30
     - 1s - loss: 2.3028 - acc: 0.0810 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 19/30
     - 1s - loss: 2.3028 - acc: 0.0770 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 20/30
     - 1s - loss: 2.3028 - acc: 0.0730 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 21/30
     - 0s - loss: 2.3028 - acc: 0.0770 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 22/30
     - 1s - loss: 2.3028 - acc: 0.0900 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 23/30
     - 0s - loss: 2.3028 - acc: 0.0700 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 24/30
     - 1s - loss: 2.3027 - acc: 0.1000 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 25/30
     - 1s - loss: 2.3027 - acc: 0.0820 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 26/30
     - 0s - loss: 2.3027 - acc: 0.0740 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 27/30
     - 1s - loss: 2.3027 - acc: 0.0770 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 28/30
     - 1s - loss: 2.3027 - acc: 0.0750 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 29/30
     - 0s - loss: 2.3027 - acc: 0.0800 - val_loss: 2.3026 - val_acc: 0.1000
    Epoch 30/30
     - 0s - loss: 2.3027 - acc: 0.0890 - val_loss: 2.3026 - val_acc: 0.1000



```python
plotly.tools.set_credentials_file(username='mgosi', api_key='X3dKXRZPub6oHfIoFii7')
trace1 = go.Scatter(
    x=epoch,
    y=learn_rate,
    name = '<b>2b2 Learning Rate</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace2 = go.Scatter(
    x=epoch,
    y=loss_train,
    name = '<b>2b2 Training Loss</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace3 = go.Scatter(
    x=epoch,
    y=loss_test,
    name = '<b>2b2 Testing Loss</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace4 = go.Scatter(
    x=epoch,
    y=history2.history["val_acc"],
    name = '<b>2b2 Testing Accuracy</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace5 = go.Scatter(
    x=epoch,
    y=history2.history["acc"],
    name = '<b>2b2 Training Accuracy</b>', # Style name/legend entry with html tags
    connectgaps=True
)
data = [trace1, trace2, trace3, trace4, trace5]
layout = dict(title = 'Task 2b2',
              xaxis = dict(title = 'Epoch'),
              yaxis = dict(title = 'Loss'),
              )
fig = dict(data=data, layout = layout)
py.iplot(fig, filename='Task 2b2')
```









```python
plotly.tools.set_credentials_file(username='mgosi', api_key='X3dKXRZPub6oHfIoFii7')
trace1 = go.Scatter(
    x=epoch,
    y=learn_rate,
    name = '<b>2b2 Learning Rate</b>', # Style name/legend entry with html tags
    connectgaps=True
)
data = [trace1]
layout = dict(title = 'Task 2b2 Learning Rate',
              xaxis = dict(title = 'Epoch'),
              yaxis = dict(title = 'Rate'),
              )
fig = dict(data=data, layout = layout)
py.iplot(fig, filename='Task 2b2 Learning Rate')
```









```python
#===One layer with L2 =====#

loss_test=[]
acc_test=[]
learn_rate=[]
loss_train=[]
acc_train=[]

model4=Task2_b3()
history4=model4.fit(XTrain, YTrain, validation_data=(XTest, YTest), nb_epoch=30, batch_size=10,
verbose=2, callbacks=callbacks_list1)
```

    /Users/akash/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:31: UserWarning:
    
    Update your `Dense` call to the Keras 2 API: `Dense(30, input_dim=784, kernel_initializer="normal", activation="sigmoid", kernel_regularizer=<keras.reg...)`
    
    /Users/akash/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:11: UserWarning:
    
    The `nb_epoch` argument in `fit` has been renamed `epochs`.
    


    _________________________________________________________________
    Layer (type)                 Output Shape              Param #   
    =================================================================
    dense_10 (Dense)             (None, 30)                23550     
    _________________________________________________________________
    dense_11 (Dense)             (None, 10)                310       
    =================================================================
    Total params: 23,860
    Trainable params: 23,860
    Non-trainable params: 0
    _________________________________________________________________
    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
     - 3s - loss: 5.2674 - acc: 0.1060 - val_loss: 2.3198 - val_acc: 0.1000
    Epoch 2/30
     - 1s - loss: 2.3200 - acc: 0.1060 - val_loss: 2.3115 - val_acc: 0.1000
    Epoch 3/30
     - 1s - loss: 2.3227 - acc: 0.0960 - val_loss: 2.3171 - val_acc: 0.1000
    Epoch 4/30
     - 1s - loss: 2.3239 - acc: 0.0880 - val_loss: 2.3057 - val_acc: 0.1000
    Epoch 5/30
     - 1s - loss: 2.3176 - acc: 0.0880 - val_loss: 2.3143 - val_acc: 0.1000
    Epoch 6/30
     - 1s - loss: 2.3165 - acc: 0.1100 - val_loss: 2.3132 - val_acc: 0.1000
    Epoch 7/30
     - 1s - loss: 2.3193 - acc: 0.0880 - val_loss: 2.3129 - val_acc: 0.1000
    Epoch 8/30
     - 1s - loss: 2.3204 - acc: 0.0800 - val_loss: 2.3111 - val_acc: 0.1490
    Epoch 9/30
     - 1s - loss: 2.3180 - acc: 0.0880 - val_loss: 2.3076 - val_acc: 0.1000
    Epoch 10/30
     - 1s - loss: 2.3159 - acc: 0.0910 - val_loss: 2.3076 - val_acc: 0.1000
    Epoch 11/30
     - 1s - loss: 2.3170 - acc: 0.0800 - val_loss: 2.3082 - val_acc: 0.1000
    Epoch 12/30
     - 0s - loss: 2.3171 - acc: 0.0760 - val_loss: 2.3065 - val_acc: 0.1000
    Epoch 13/30
     - 0s - loss: 2.3149 - acc: 0.0880 - val_loss: 2.3080 - val_acc: 0.1000
    Epoch 14/30
     - 0s - loss: 2.3162 - acc: 0.0800 - val_loss: 2.3068 - val_acc: 0.1000
    Epoch 15/30
     - 0s - loss: 2.3097 - acc: 0.0910 - val_loss: 2.3126 - val_acc: 0.1000
    Epoch 16/30
     - 1s - loss: 2.3136 - acc: 0.0950 - val_loss: 2.3061 - val_acc: 0.1000
    Epoch 17/30
     - 0s - loss: 2.3152 - acc: 0.0840 - val_loss: 2.3041 - val_acc: 0.1000
    Epoch 18/30
     - 0s - loss: 2.3132 - acc: 0.0790 - val_loss: 2.3058 - val_acc: 0.1000
    Epoch 19/30
     - 1s - loss: 2.3121 - acc: 0.0880 - val_loss: 2.3055 - val_acc: 0.1000
    Epoch 20/30
     - 0s - loss: 2.3109 - acc: 0.0870 - val_loss: 2.3072 - val_acc: 0.1000
    Epoch 21/30
     - 1s - loss: 2.3127 - acc: 0.0860 - val_loss: 2.3043 - val_acc: 0.1000
    Epoch 22/30
     - 1s - loss: 2.3093 - acc: 0.0970 - val_loss: 2.3081 - val_acc: 0.1000
    Epoch 23/30
     - 1s - loss: 2.3107 - acc: 0.0940 - val_loss: 2.3054 - val_acc: 0.1840
    Epoch 24/30
     - 1s - loss: 2.3110 - acc: 0.1020 - val_loss: 2.3038 - val_acc: 0.1000
    Epoch 25/30
     - 0s - loss: 2.3119 - acc: 0.0830 - val_loss: 2.3042 - val_acc: 0.1000
    Epoch 26/30
     - 0s - loss: 2.3108 - acc: 0.0870 - val_loss: 2.3039 - val_acc: 0.1000
    Epoch 27/30
     - 1s - loss: 2.3099 - acc: 0.0870 - val_loss: 2.3051 - val_acc: 0.1000
    Epoch 28/30
     - 1s - loss: 2.3106 - acc: 0.0810 - val_loss: 2.3047 - val_acc: 0.1000
    Epoch 29/30
     - 1s - loss: 2.3110 - acc: 0.0870 - val_loss: 2.3043 - val_acc: 0.1000
    Epoch 30/30
     - 1s - loss: 2.3109 - acc: 0.0860 - val_loss: 2.3032 - val_acc: 0.1000



```python
plotly.tools.set_credentials_file(username='mgosi', api_key='X3dKXRZPub6oHfIoFii7')
trace1 = go.Scatter(
    x=epoch,
    y=learn_rate,
    name = '<b>2b3 Learning Rate</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace2 = go.Scatter(
    x=epoch,
    y=loss_train,
    name = '<b>2b3 Training Loss</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace3 = go.Scatter(
    x=epoch,
    y=loss_test,
    name = '<b>2b3 Testing Loss</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace4 = go.Scatter(
    x=epoch,
    y=history2.history["val_acc"],
    name = '<b>2b3 Testing Accuracy</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace5 = go.Scatter(
    x=epoch,
    y=history2.history["acc"],
    name = '<b>2b3 Training Accuracy</b>', # Style name/legend entry with html tags
    connectgaps=True
)
data = [trace1, trace2, trace3, trace4, trace5]
layout = dict(title = 'Task 2b3',
              xaxis = dict(title = 'Epoch'),
              yaxis = dict(title = 'Loss'),
              )
fig = dict(data=data, layout = layout)
py.iplot(fig, filename='Task 2b3')
```









```python
plotly.tools.set_credentials_file(username='mgosi', api_key='X3dKXRZPub6oHfIoFii7')
trace1 = go.Scatter(
    x=epoch,
    y=learn_rate,
    name = '<b>2b3 Learning Rate</b>', # Style name/legend entry with html tags
    connectgaps=True
)
data = [trace1]
layout = dict(title = 'Task 2b3 Learning Rate',
              xaxis = dict(title = 'Epoch'),
              yaxis = dict(title = 'Rate'),
              )
fig = dict(data=data, layout = layout)
py.iplot(fig, filename='Task 2b3 Learning Rate')
```









```python
#=== Two layers with L2 =====#

loss_test=[]
acc_test=[]
learn_rate=[]
loss_train=[]
acc_train=[]

model5=Task2_b4()
history5=model5.fit(XTrain, YTrain, validation_data=(XTest, YTest), nb_epoch=30, batch_size=10,
verbose=2, callbacks=callbacks_list1)
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #   
    =================================================================
    dense_12 (Dense)             (None, 30)                23550     
    _________________________________________________________________
    dense_13 (Dense)             (None, 30)                930       
    _________________________________________________________________
    dense_14 (Dense)             (None, 10)                310       
    =================================================================
    Total params: 24,790
    Trainable params: 24,790
    Non-trainable params: 0
    _________________________________________________________________


    /Users/akash/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:39: UserWarning:
    
    Update your `Dense` call to the Keras 2 API: `Dense(30, input_dim=784, kernel_initializer="normal", activation="sigmoid", kernel_regularizer=<keras.reg...)`
    
    /Users/akash/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:40: UserWarning:
    
    Update your `Dense` call to the Keras 2 API: `Dense(30, kernel_initializer="normal", activation="sigmoid", kernel_regularizer=<keras.reg...)`
    
    /Users/akash/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:11: UserWarning:
    
    The `nb_epoch` argument in `fit` has been renamed `epochs`.
    


    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
     - 3s - loss: 5.3804 - acc: 0.0850 - val_loss: 2.3171 - val_acc: 0.1000
    Epoch 2/30
     - 1s - loss: 2.3292 - acc: 0.0820 - val_loss: 2.3065 - val_acc: 0.1000
    Epoch 3/30
     - 1s - loss: 2.3155 - acc: 0.1020 - val_loss: 2.3544 - val_acc: 0.1000
    Epoch 4/30
     - 1s - loss: 2.3237 - acc: 0.1080 - val_loss: 2.3151 - val_acc: 0.1000
    Epoch 5/30
     - 0s - loss: 2.3276 - acc: 0.0950 - val_loss: 2.3146 - val_acc: 0.1000
    Epoch 6/30
     - 1s - loss: 2.3227 - acc: 0.0940 - val_loss: 2.3173 - val_acc: 0.1000
    Epoch 7/30
     - 0s - loss: 2.3273 - acc: 0.0810 - val_loss: 2.3185 - val_acc: 0.1000
    Epoch 8/30
     - 0s - loss: 2.3272 - acc: 0.0910 - val_loss: 2.3165 - val_acc: 0.1000
    Epoch 9/30
     - 1s - loss: 2.3256 - acc: 0.0930 - val_loss: 2.3154 - val_acc: 0.1000
    Epoch 10/30
     - 0s - loss: 2.3240 - acc: 0.1020 - val_loss: 2.3148 - val_acc: 0.1000
    Epoch 11/30
     - 1s - loss: 2.3274 - acc: 0.0800 - val_loss: 2.3228 - val_acc: 0.1000
    Epoch 12/30
     - 0s - loss: 2.3251 - acc: 0.0940 - val_loss: 2.3230 - val_acc: 0.1000
    Epoch 13/30
     - 0s - loss: 2.3260 - acc: 0.0980 - val_loss: 2.3254 - val_acc: 0.1000
    Epoch 14/30
     - 0s - loss: 2.3283 - acc: 0.0850 - val_loss: 2.3106 - val_acc: 0.1000
    Epoch 15/30
     - 0s - loss: 2.3213 - acc: 0.1040 - val_loss: 2.3202 - val_acc: 0.1000
    Epoch 16/30
     - 1s - loss: 2.3247 - acc: 0.0950 - val_loss: 2.3126 - val_acc: 0.1000
    Epoch 17/30
     - 0s - loss: 2.3240 - acc: 0.0900 - val_loss: 2.3117 - val_acc: 0.1000
    Epoch 18/30
     - 0s - loss: 2.3271 - acc: 0.0780 - val_loss: 2.3134 - val_acc: 0.1000
    Epoch 19/30
     - 0s - loss: 2.3268 - acc: 0.0770 - val_loss: 2.3088 - val_acc: 0.1000
    Epoch 20/30
     - 0s - loss: 2.3225 - acc: 0.0930 - val_loss: 2.3226 - val_acc: 0.1000
    Epoch 21/30
     - 1s - loss: 2.3228 - acc: 0.0930 - val_loss: 2.3160 - val_acc: 0.1000
    Epoch 22/30
     - 0s - loss: 2.3247 - acc: 0.0890 - val_loss: 2.3228 - val_acc: 0.1000
    Epoch 23/30
     - 0s - loss: 2.3247 - acc: 0.0930 - val_loss: 2.3112 - val_acc: 0.1000
    Epoch 24/30
     - 0s - loss: 2.3259 - acc: 0.0920 - val_loss: 2.3071 - val_acc: 0.1000
    Epoch 25/30
     - 0s - loss: 2.3242 - acc: 0.1020 - val_loss: 2.3113 - val_acc: 0.1000
    Epoch 26/30
     - 1s - loss: 2.3283 - acc: 0.0930 - val_loss: 2.3166 - val_acc: 0.1000
    Epoch 27/30
     - 0s - loss: 2.3226 - acc: 0.1060 - val_loss: 2.3087 - val_acc: 0.1000
    Epoch 28/30
     - 1s - loss: 2.3235 - acc: 0.0920 - val_loss: 2.3129 - val_acc: 0.1000
    Epoch 29/30
     - 0s - loss: 2.3251 - acc: 0.0840 - val_loss: 2.3188 - val_acc: 0.1000
    Epoch 30/30
     - 1s - loss: 2.3246 - acc: 0.0960 - val_loss: 2.3139 - val_acc: 0.1000



```python
plotly.tools.set_credentials_file(username='mgosi', api_key='X3dKXRZPub6oHfIoFii7')
trace1 = go.Scatter(
    x=epoch,
    y=learn_rate,
    name = '<b>2b4 Learning Rate</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace2 = go.Scatter(
    x=epoch,
    y=loss_train,
    name = '<b>2b4 Training Loss</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace3 = go.Scatter(
    x=epoch,
    y=loss_test,
    name = '<b>2b4 Testing Loss</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace4 = go.Scatter(
    x=epoch,
    y=history2.history["val_acc"],
    name = '<b>2b4 Testing Accuracy</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace5 = go.Scatter(
    x=epoch,
    y=history2.history["acc"],
    name = '<b>2b4 Training Accuracy</b>', # Style name/legend entry with html tags
    connectgaps=True
)
data = [trace1, trace2, trace3, trace4, trace5]
layout = dict(title = 'Task 2b4',
              xaxis = dict(title = 'Epoch'),
              yaxis = dict(title = 'Loss'),
              )
fig = dict(data=data, layout = layout)
py.iplot(fig, filename='Task 2b4')
```









```python
plotly.tools.set_credentials_file(username='mgosi', api_key='X3dKXRZPub6oHfIoFii7')
trace1 = go.Scatter(
    x=epoch,
    y=learn_rate,
    name = '<b>2b4 Learning Rate</b>', # Style name/legend entry with html tags
    connectgaps=True
)
data = [trace1]
layout = dict(title = 'Task 2b4 Learning Rate',
              xaxis = dict(title = 'Epoch'),
              yaxis = dict(title = 'Rate'),
              )
fig = dict(data=data, layout = layout)
py.iplot(fig, filename='Task 2b4 Learning Rate')
```









```python

```


```python
#===== THree hidden layers with L2 ======#
loss_test=[]
acc_test=[]
learn_rate=[]
loss_train=[]
acc_train=[]

model6=Task2_b5()
history6=model6.fit(XTrain, YTrain, validation_data=(XTest, YTest), nb_epoch=30, batch_size=10,
verbose=2, callbacks=callbacks_list1)
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #   
    =================================================================
    dense_15 (Dense)             (None, 30)                23550     
    _________________________________________________________________
    dense_16 (Dense)             (None, 30)                930       
    _________________________________________________________________
    dense_17 (Dense)             (None, 30)                930       
    _________________________________________________________________
    dense_18 (Dense)             (None, 10)                310       
    =================================================================
    Total params: 25,720
    Trainable params: 25,720
    Non-trainable params: 0
    _________________________________________________________________


    /Users/akash/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:48: UserWarning:
    
    Update your `Dense` call to the Keras 2 API: `Dense(30, input_dim=784, kernel_initializer="normal", activation="sigmoid", kernel_regularizer=<keras.reg...)`
    
    /Users/akash/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:49: UserWarning:
    
    Update your `Dense` call to the Keras 2 API: `Dense(30, kernel_initializer="normal", activation="sigmoid", kernel_regularizer=<keras.reg...)`
    
    /Users/akash/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:50: UserWarning:
    
    Update your `Dense` call to the Keras 2 API: `Dense(30, kernel_initializer="normal", activation="sigmoid", kernel_regularizer=<keras.reg...)`
    
    /Users/akash/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:10: UserWarning:
    
    The `nb_epoch` argument in `fit` has been renamed `epochs`.
    


    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
     - 3s - loss: 5.4819 - acc: 0.0920 - val_loss: 2.3197 - val_acc: 0.1000
    Epoch 2/30
     - 1s - loss: 2.3274 - acc: 0.0800 - val_loss: 2.3092 - val_acc: 0.1000
    Epoch 3/30
     - 1s - loss: 2.3290 - acc: 0.0920 - val_loss: 2.3083 - val_acc: 0.1000
    Epoch 4/30
     - 1s - loss: 2.3224 - acc: 0.0850 - val_loss: 2.3127 - val_acc: 0.1000
    Epoch 5/30
     - 1s - loss: 2.3216 - acc: 0.0920 - val_loss: 2.3157 - val_acc: 0.1000
    Epoch 6/30
     - 1s - loss: 2.3251 - acc: 0.0960 - val_loss: 2.3095 - val_acc: 0.1000
    Epoch 7/30
     - 1s - loss: 2.3206 - acc: 0.0880 - val_loss: 2.3239 - val_acc: 0.1000
    Epoch 8/30
     - 1s - loss: 2.3233 - acc: 0.0950 - val_loss: 2.3092 - val_acc: 0.1000
    Epoch 9/30
     - 1s - loss: 2.3204 - acc: 0.0840 - val_loss: 2.3229 - val_acc: 0.1000
    Epoch 10/30
     - 1s - loss: 2.3251 - acc: 0.0960 - val_loss: 2.3109 - val_acc: 0.1000
    Epoch 11/30
     - 1s - loss: 2.3147 - acc: 0.0970 - val_loss: 2.3421 - val_acc: 0.1000
    Epoch 12/30
     - 1s - loss: 2.3260 - acc: 0.0890 - val_loss: 2.3102 - val_acc: 0.1000
    Epoch 13/30
     - 1s - loss: 2.3182 - acc: 0.0980 - val_loss: 2.3139 - val_acc: 0.1000
    Epoch 14/30
     - 1s - loss: 2.3272 - acc: 0.0920 - val_loss: 2.3087 - val_acc: 0.1000
    Epoch 15/30
     - 1s - loss: 2.3248 - acc: 0.0830 - val_loss: 2.3134 - val_acc: 0.1000
    Epoch 16/30
     - 1s - loss: 2.3193 - acc: 0.0780 - val_loss: 2.3180 - val_acc: 0.1000
    Epoch 17/30
     - 1s - loss: 2.3247 - acc: 0.0950 - val_loss: 2.3073 - val_acc: 0.1000
    Epoch 18/30
     - 1s - loss: 2.3226 - acc: 0.0870 - val_loss: 2.3148 - val_acc: 0.1000
    Epoch 19/30
     - 1s - loss: 2.3165 - acc: 0.0900 - val_loss: 2.3149 - val_acc: 0.1000
    Epoch 20/30
     - 1s - loss: 2.3229 - acc: 0.0850 - val_loss: 2.3177 - val_acc: 0.1000
    Epoch 21/30
     - 1s - loss: 2.3203 - acc: 0.1020 - val_loss: 2.3158 - val_acc: 0.1000
    Epoch 22/30
     - 1s - loss: 2.3216 - acc: 0.0760 - val_loss: 2.3171 - val_acc: 0.1000
    Epoch 23/30
     - 1s - loss: 2.3202 - acc: 0.0890 - val_loss: 2.3123 - val_acc: 0.1000
    Epoch 24/30
     - 1s - loss: 2.3225 - acc: 0.0860 - val_loss: 2.3104 - val_acc: 0.1000
    Epoch 25/30
     - 1s - loss: 2.3168 - acc: 0.0960 - val_loss: 2.3210 - val_acc: 0.1000
    Epoch 26/30
     - 1s - loss: 2.3213 - acc: 0.0940 - val_loss: 2.3072 - val_acc: 0.1000
    Epoch 27/30
     - 1s - loss: 2.3188 - acc: 0.1050 - val_loss: 2.3062 - val_acc: 0.1000
    Epoch 28/30
     - 1s - loss: 2.3208 - acc: 0.0830 - val_loss: 2.3113 - val_acc: 0.1000
    Epoch 29/30
     - 2s - loss: 2.3196 - acc: 0.1000 - val_loss: 2.3109 - val_acc: 0.1000
    Epoch 30/30
     - 1s - loss: 2.3228 - acc: 0.0930 - val_loss: 2.3125 - val_acc: 0.1000



```python
plotly.tools.set_credentials_file(username='mgosi', api_key='X3dKXRZPub6oHfIoFii7')
trace1 = go.Scatter(
    x=epoch,
    y=learn_rate,
    name = '<b>2b5 Learning Rate</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace2 = go.Scatter(
    x=epoch,
    y=loss_train,
    name = '<b>2b5 Training Loss</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace3 = go.Scatter(
    x=epoch,
    y=loss_test,
    name = '<b>2b5 Testing Loss</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace4 = go.Scatter(
    x=epoch,
    y=history2.history["val_acc"],
    name = '<b>2b5 Testing Accuracy</b>', # Style name/legend entry with html tags
    connectgaps=True
)
trace5 = go.Scatter(
    x=epoch,
    y=history2.history["acc"],
    name = '<b>2b5 Training Accuracy</b>', # Style name/legend entry with html tags
    connectgaps=True
)
data = [trace1, trace2, trace3, trace4, trace5]
layout = dict(title = 'Task 2b5',
              xaxis = dict(title = 'Epoch'),
              yaxis = dict(title = 'Loss'),
              )
fig = dict(data=data, layout = layout)
py.iplot(fig, filename='Task 2b5')
```









```python
plotly.tools.set_credentials_file(username='mgosi', api_key='X3dKXRZPub6oHfIoFii7')
trace1 = go.Scatter(
    x=epoch,
    y=learn_rate,
    name = '<b>2b5 Learning Rate</b>', # Style name/legend entry with html tags
    connectgaps=True
)
data = [trace1]
layout = dict(title = 'Task 2b5 Learning Rate',
              xaxis = dict(title = 'Epoch'),
              yaxis = dict(title = 'Rate'),
              )
fig = dict(data=data, layout = layout)
py.iplot(fig, filename='Task 2b5 Learning Rate')
```









```python
scores1 = model1.evaluate(XTest, YTest, verbose=0)
scores2 = model2.evaluate(XTest, YTest, verbose=0)
scores3 = model3.evaluate(XTest, YTest, verbose=0)
scores4 = model4.evaluate(XTest, YTest, verbose=0)
scores5 = model5.evaluate(XTest, YTest, verbose=0)
scores6 = model6.evaluate(XTest, YTest, verbose=0)
```


```python
print("Loss and Accuracy of the models are as follows :")
print (scores1)
print (scores2)
print (scores3)
print (scores4)
print (scores5)
print (scores6)
```

    Loss and Accuracy of the models are as follows :
    [0.5465954859256744, 0.834]
    [0.8550777387619019, 0.737]
    [2.3025833435058596, 0.1]
    [2.30324348449707, 0.1]
    [2.3139245147705076, 0.1]
    [2.3124769325256347, 0.1]




## Task 2c


```python
from scipy.ndimage import rotate
from scipy.misc import face
from keras.layers.convolutional import Conv2D
from keras.layers.convolutional import MaxPooling2D
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.layers import Flatten
from keras.utils import np_utils
from keras.optimizers import SGD

(X_train, y_train), (X_test, y_test) = mnist.load_data()
```


```python
#This is used to rotate the images by the desired amount. 
#It also shifts the pixels in 8 directions.i.e : Right, Left, Top, Down, and all four Diagonals
def imgaugment(angle):
    
    for i in range(0,60000):
    #anticlockwise rotation
        rot = rotate(X_train[i], angle=angle, reshape=False)

        #left shift
        for k in range(14,17):
            temp=rot[k][0]
            for l in range(27):
                rot[k][l]=rot[k][l+1]
            rot[k][27]=temp
        #right shift    
        for k in range(22,25):
            temp=rot[k][27]
            for l in range(27):
                rot[k][27-l]=rot[k][27-l-1]
            rot[k][0]=temp
        #top shift    
        for k in range(17,20):
            temp=rot[0][k]
            for l in range(27):
                rot[l][k]=rot[l+1][k]
            rot[27][k]=temp
        #bottom shift    
        for k in range(14,17):
            temp=rot[27][k]
            for l in range(27):
                rot[27-l][k]=rot[27-l-1][k]
            rot[0][k]=temp

        #second diagonal shift left
        b=[]
        for i in range(0,28):
            b.append(rot[27-i][i])

        temp=b[0]

        for i in range(0,5):
            b[i]=b[i+1]
        b[5]=temp

        for i in range(0,28):
            rot[27-i][i]=b[i]

        #first diagonal shift left
        c=[]

        for i in range(0,28):
            c.append(rot[i][i])


        temp=c[0]
        for i in range(0,27):
            c[i]=c[i+1]
        c[27]=temp


        for i in range(0,28):
            rot[i][i]=c[i]

        #second diagonal shift right
        temp=b[27]
        for i in range(0,27):
            b[27-i]=b[27-i-1]
        b[0]=temp


        for i in range(0,28):
            rot[27-i][i]=b[i]

        #first diagonal rotation right

        temp=c[27]
        for i in range(0,27):
            c[27-i]=c[27-i-1]
        c[0]=temp


        for i in range(0,28):
            rot[i][i]=c[i]


        #plt.imshow(rot.reshape(28,28))

        #plt.show()
        X_train[i]=rot[i]


    for i in range(0,10000):

        rot1 = rotate(X_test[i], angle=angle, reshape=False)

        #left shift
        for k in range(22,25):
            temp=rot1[k][0]
            for l in range(27):
                rot1[k][l]=rot1[k][l+1]
            rot1[k][27]=temp
        #right shift    
        for k in range(22,25):
            temp=rot1[k][27]
            for l in range(27):
                rot1[k][27-l]=rot1[k][27-l-1]
            rot1[k][0]=temp
        #top shift    
        for k in range(22,25):
            temp=rot1[0][k]
            for l in range(27):
                rot1[l][k]=rot1[l+1][k]
            rot1[27][k]=temp
        #bottom shift    
        for k in range(22,25):
            temp=rot1[27][k]
            for l in range(27):
                rot1[27-l][k]=rot1[27-l-1][k]
            rot1[0][k]=temp

        #second diagonal shift left
        b=[]
        for i in range(0,28):
            b.append(rot1[27-i][i])

        temp=b[0]

        for i in range(0,5):
            b[i]=b[i+1]
        b[5]=temp

        for i in range(0,28):
            rot1[27-i][i]=b[i]

        #second diagonal shift right
        temp=b[27]
        for i in range(0,27):
            b[27-i]=b[27-i-1]
        b[0]=temp


        for i in range(0,28):
            rot1[27-i][i]=b[i]

        #first diagonal shift left
        c=[]

        for i in range(0,28):
            c.append(rot1[i][i])


        temp=c[0]
        for i in range(0,27):
            c[i]=c[i+1]
        c[27]=temp


        for i in range(0,28):
            rot1[i][i]=c[i]

        #first diagonal shift right

        temp=c[27]
        for i in range(0,27):
            c[27-i]=c[27-i-1]
        c[0]=temp


        for i in range(0,28):
            rot1[i][i]=c[i]


        #plt.imshow(rot1.reshape(28,28))

        #plt.show()
        X_test[i]=rot1[i]
```


```python

```


```python
imgaugment(2)
X_train = X_train.reshape(X_train.shape[0], 1, 28, 28).astype('float32')
X_test = X_test.reshape(X_test.shape[0], 1, 28, 28).astype('float32')
```


```python
XTrain.shape
```




    (1000, 1, 28, 28)




```python
XTrain = X_train[y_train[:,]==0][:100]
YTrain = np.zeros(100).tolist()
XTest = X_test[y_test[:,]==0][:100]
YTest = np.zeros(100).tolist()
YTemp = np.empty(100)
for i in range(1,10):
    XTrain = np.concatenate((XTrain, X_train[y_train[:,]==i][:100]))
    YTemp.fill(i)
    YTrain = YTrain + YTemp.tolist()
    XTest = np.concatenate((XTest, X_test[y_test[:,]==i][:100] ))
    YTest = YTest + YTemp.tolist()

print (XTrain.shape)
XTrain = XTrain/255
XTest = XTest/255
YTrain = np_utils.to_categorical(YTrain)
YTest = np_utils.to_categorical(YTest)
XTrain.shape, XTest.shape, YTrain.shape, YTest.shape

numOfClasses = YTest.shape[1]
```

    (1000, 1, 28, 28)



```python
def task3_1():
    model = Sequential()
    model.add(Conv2D(32, (5, 5), input_shape=(1, 28, 28), activation='relu', data_format="channels_first"))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Flatten())
    model.add(Dense(128, activation='relu'))
    model.add(Dense(numOfClasses, activation='softmax'))
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model

def task3_2():
    model = Sequential()
    model.add(Conv2D(32, (5, 5), input_shape=(1, 28, 28), activation='relu', data_format = "channels_first"))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.2))
    model.add(Flatten())
    model.add(Dense(128, activation='relu'))
    model.add(Dense(numOfClasses, activation='softmax'))
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model
```


```python
model1= task3_1()
history1=model1.fit(XTrain, YTrain, validation_data=(XTest, YTest), epochs=10, batch_size=200, verbose=2)
```

    WARNING:tensorflow:From /Users/akash/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.cast instead.
    Train on 1000 samples, validate on 1000 samples
    Epoch 1/10



```python
#===== With Dropout ====#
model2= task3_2()
history2=model2.fit(XTrain, YTrain, validation_data=(XTest, YTest), epochs=10, batch_size=200, verbose=2)
```


```python

```

c6bc1d1e6e11b59228fd26000fb86a34fe1a3d36
ipynb
Jupyter Notebook
Assignment 4.ipynb
Mgosi/Pattern-Recognition
e4a51b41e3ac0e64456adb629da2e8d8825c6b12
MIT
Assignment 4.ipynb
Mgosi/Pattern-Recognition
e4a51b41e3ac0e64456adb629da2e8d8825c6b12
MIT
Assignment 4.ipynb
Mgosi/Pattern-Recognition
e4a51b41e3ac0e64456adb629da2e8d8825c6b12
MIT
Qwen/Qwen-72B
1. YES
2. YES