# Intro to Pattern Recognition
# Problem Set 4: Neural Networks
### M S Mohamed Fazil
### UB Id : mm549

## Introduction
We will be using neural networks to build a model that classifies hand written digits using thr MNIST database. The MNIST database (Modified National Institute of Standards and Technology Database) is a large collection of handwritten digits which happens to be a state of the art data set to train and test machine learning models using image processing.It consist of 60000 Training samples and 10000 Testing Samples. Each sample image consist of normalized $28X28$ pixels grayscale image stored as 784 dimensional vector for each sample.This is a way of dimension reduction done by the MNIST to use it for model developments.

### Libraries Used
For loading the given MNIST datasets we will be using the mnist library to fetch the data.The MNIST() function will be used to fetch the datasets from the directory './data'. For the purpose of building neural networks and trainning we will be using Keras library which is backed by the Tensorflow. We will use OpenCv for the image processing applications.


```python
from keras.models import Sequential
from keras.layers import Dense, Activation, Conv2D, MaxPooling2D, Dropout, Flatten
from keras.callbacks import LambdaCallback,ModelCheckpoint
from keras.utils import to_categorical
from keras import optimizers,regularizers
import numpy as np
import cv2
import os
from mnist.loader import MNIST
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import random as rd
os.environ["CUDA_VISIBLE_DEVICES"]="1"
%matplotlib inline

m = MNIST('./data')
```

    Using TensorFlow backend.


## 1. Neural Network to Maximize the log Likehood 

### Minimization of Negative Log Likehood with Softmax Output Layer
Softmax is an activation function of a node which is commonly placed as the output layer of a Neural Network. It is mostly used in multiclass classification problems. The Function is defined as 
\begin{equation}
S(V_i) = \frac{e^{V_i}}{\sum_je^{V_i}}\
\end{equation}
The softmax function squashes the input vector values between 0 to 1 and because of the normalization of exponential , the sum of the values in the vector tends to be 1. The output values of a Softmax function can be interpretted as a probabilities of the different labels present in the multiclass.In our case the Softmax outputs a vector of 10 values corresponding to the class labels.
Negative Log-Likehood is a loss function (aka criterion function) which is used with the Softmax activation function to minimize the criterion function. It is given as 
\begin{equation}
J_0 = -log\,p(D;w) = -log\,p(\{(x_n,t_n):n=1,2,4,...\};w)
\end{equation}
\begin{equation}
J_0 =-log\prod_n\prod_{m=0}^9p(t_n=m|x_n;w)
\end{equation}
To minimize the criterion function we must differentiate the softmax function with respect to the Negative Log Likehood function.
### Neural Network to Maximize the Posterior Likehood by Minimizing Criterion Function with L2 Regularization
Let us assume that a Gaussion Prior of the weight distribution is given with the $\alpha$.
\begin{equation}
p(w;\alpha)=N(N,\alpha l)
\end{equation}
Regularization is used to avoid overfitting of the traning datasets into the model.Therefore depending on the gradient of the weights between the epochs a part of the weights is subtracted by the regularization component determined with the regularization parameter $\alpha$. The Gaussian Prior function can be used in par with the L2 regularization during the minimization of the Criterion function which will maximize the Posterior Likehood. The combined loss function can be given as 
\begin{equation}
J(w) = J_0(w)-log\,p(w;\alpha^{-1})
\end{equation}

## Preparing the Datasets
We prepare a dataset which will consist of 1000 training samples with labels where 100 images for each digit and another 1000 testing samples with label where 100 images for each digit. The target consist of a class with ten classification labels.


```python
classes = [0,1,2,3,4,5,6,7,8,9]
xtrain,ytrain = m.load_training()
xtest,ytest = m.load_testing()

# Divide the given datasets to 1000 training dataset and 1000 testing dataset
xtrain = xtrain[0:1000]
ytrain = ytrain[0:1000]
xtest = xtest[0:1000]
ytest = ytest[0:1000]
xtrain = np.asarray(xtrain).astype(np.float32)
ytrain = np.asarray(ytrain).astype(np.float32)
xtest = np.asarray(xtest).astype(np.float32)
ytest = np.asarray(ytest).astype(np.float32)

n_classes = len(classes)
#0-1 Hot encoding
label_train = np.zeros((ytrain.shape[0], n_classes))
a = np.arange(ytrain.shape[0], dtype=np.int64)
b = np.array(ytrain, dtype=np.int64).reshape((ytrain.shape[0],))
label_train[a, b] = 1
label_test = np.zeros((ytest.shape[0], n_classes))
c = np.arange(ytest.shape[0], dtype=np.int64)
d = np.array(ytest, dtype=np.int64).reshape((ytest.shape[0],))
label_test[c, d] = 1
```

### Function to Plot the Results of a Training Model
This fuction plots the Traing and Testing's loss,criterion function, accuracy and the learning speed of each hidden layer with respect to each epoch during training phase. The learning Speed is the measure of the speed of change of weights after every epoch.


```python
def plot_model_result(history,weights,final_w):
    plt.plot(history.history['loss'], label='Training data')
    plt.plot(history.history['val_loss'], label='Testing data')
    plt.legend(['train', 'test'], loc='upper left')
    plt.title('Loss of Training and Testing Data (Criterion Function)')
    plt.ylabel('MAE value')
    plt.xlabel('No. epoch')
    plt.show()
    plt.plot(history.history['accuracy'], label='Training data')
    plt.plot(history.history['val_accuracy'], label='Testing data')
    plt.legend(['train', 'test'], loc='upper left')
    plt.title('Accuracy of Training and Testing Data')
    plt.ylabel('Accuracy value')
    plt.xlabel('No. epoch')
    plt.show()
    cnt = 0
    '''
    Plot the Learning Speed of the Model
    '''
    np.seterr(divide='ignore', invalid='ignore')
    for i in range(0,len(weights)):
        w = weights[i]
        W = final_w[i]
        learning_speed = [0]
        for i in range(1,len(w)):
            wold = w[i-1]
            wnew = w[i]
            diff = wold-wnew
            d = wnew
            m = np.median(d[d>0])
            d[d==0] = m
            diff = abs(diff)/d
            if diff.size:
                avg = abs(np.mean(diff))
            else:
                avg = 0
            learning_speed.append(avg)
        plt.plot(learning_speed, label='Learning Speed')
        cnt = cnt+1
        plt.title('Learning Speed of Hidden Layer {}'.format(cnt))
        plt.ylabel('Learning Speed')
        plt.xlabel('No. epoch')
        plt.show()
        
    print('Training Accuracy -',history.history['accuracy'][29])
    print('Training Loss -',history.history['loss'][29])
    print('Testing Accuracy -',history.history['val_accuracy'][29])
    print('Testing Loss -',history.history['val_loss'][29])
```

## 2(a). Neural Network with 1 Hidden Layer without regularization


```python
model = Sequential()
model.add(Dense(30, activation='sigmoid', input_dim=xtrain.shape[1]))
model.add(Dense(10, activation='softmax'))
weights = []
save_weights = LambdaCallback(on_epoch_end=lambda batch, logs: weights.append(model.layers[0].get_weights()[0]))
sgd = optimizers.SGD(learning_rate=0.1)
model.compile(optimizer=sgd,
              loss='binary_crossentropy',
              metrics=['accuracy'])
history = model.fit(xtrain, label_train,validation_data=(xtest, label_test), 
                    epochs=30, batch_size=10, shuffle = False,callbacks = [save_weights])
final_w = [model.layers[0].get_weights()[0]]
weights = [weights]
```

    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
    1000/1000 [==============================] - 1s 505us/step - loss: 0.3249 - accuracy: 0.8999 - val_loss: 0.3080 - val_accuracy: 0.9000
    Epoch 2/30
    1000/1000 [==============================] - 0s 323us/step - loss: 0.2965 - accuracy: 0.9000 - val_loss: 0.2910 - val_accuracy: 0.9000
    Epoch 3/30
    1000/1000 [==============================] - 0s 316us/step - loss: 0.2822 - accuracy: 0.9000 - val_loss: 0.2849 - val_accuracy: 0.9000
    Epoch 4/30
    1000/1000 [==============================] - 0s 315us/step - loss: 0.2640 - accuracy: 0.9015 - val_loss: 0.2661 - val_accuracy: 0.9001
    Epoch 5/30
    1000/1000 [==============================] - 0s 317us/step - loss: 0.2475 - accuracy: 0.9041 - val_loss: 0.2513 - val_accuracy: 0.9071
    Epoch 6/30
    1000/1000 [==============================] - 0s 316us/step - loss: 0.2381 - accuracy: 0.9070 - val_loss: 0.2443 - val_accuracy: 0.9095
    Epoch 7/30
    1000/1000 [==============================] - 0s 318us/step - loss: 0.2319 - accuracy: 0.9082 - val_loss: 0.2372 - val_accuracy: 0.9092
    Epoch 8/30
    1000/1000 [==============================] - 0s 315us/step - loss: 0.2224 - accuracy: 0.9087 - val_loss: 0.2269 - val_accuracy: 0.9104
    Epoch 9/30
    1000/1000 [==============================] - 0s 313us/step - loss: 0.2101 - accuracy: 0.9119 - val_loss: 0.2213 - val_accuracy: 0.9106
    Epoch 10/30
    1000/1000 [==============================] - 0s 312us/step - loss: 0.2048 - accuracy: 0.9126 - val_loss: 0.2174 - val_accuracy: 0.9133
    Epoch 11/30
    1000/1000 [==============================] - 0s 319us/step - loss: 0.2020 - accuracy: 0.9161 - val_loss: 0.2135 - val_accuracy: 0.9161
    Epoch 12/30
    1000/1000 [==============================] - 0s 314us/step - loss: 0.1939 - accuracy: 0.9190 - val_loss: 0.2101 - val_accuracy: 0.9159
    Epoch 13/30
    1000/1000 [==============================] - 0s 318us/step - loss: 0.1959 - accuracy: 0.9158 - val_loss: 0.2097 - val_accuracy: 0.9150
    Epoch 14/30
    1000/1000 [==============================] - 0s 311us/step - loss: 0.1902 - accuracy: 0.9178 - val_loss: 0.1968 - val_accuracy: 0.9169
    Epoch 15/30
    1000/1000 [==============================] - 0s 310us/step - loss: 0.1833 - accuracy: 0.9249 - val_loss: 0.2056 - val_accuracy: 0.9175
    Epoch 16/30
    1000/1000 [==============================] - 0s 314us/step - loss: 0.1747 - accuracy: 0.9285 - val_loss: 0.2009 - val_accuracy: 0.9232
    Epoch 17/30
    1000/1000 [==============================] - 0s 319us/step - loss: 0.1899 - accuracy: 0.9228 - val_loss: 0.2166 - val_accuracy: 0.9153
    Epoch 18/30
    1000/1000 [==============================] - 0s 319us/step - loss: 0.1925 - accuracy: 0.9206 - val_loss: 0.2165 - val_accuracy: 0.9113
    Epoch 19/30
    1000/1000 [==============================] - 0s 319us/step - loss: 0.1996 - accuracy: 0.9146 - val_loss: 0.2226 - val_accuracy: 0.9101
    Epoch 20/30
    1000/1000 [==============================] - 0s 316us/step - loss: 0.1864 - accuracy: 0.9238 - val_loss: 0.2153 - val_accuracy: 0.9166
    Epoch 21/30
    1000/1000 [==============================] - 0s 315us/step - loss: 0.1847 - accuracy: 0.9255 - val_loss: 0.1938 - val_accuracy: 0.9202
    Epoch 22/30
    1000/1000 [==============================] - 0s 314us/step - loss: 0.1612 - accuracy: 0.9370 - val_loss: 0.1815 - val_accuracy: 0.9279
    Epoch 23/30
    1000/1000 [==============================] - 0s 318us/step - loss: 0.1527 - accuracy: 0.9420 - val_loss: 0.1809 - val_accuracy: 0.9278
    Epoch 24/30
    1000/1000 [==============================] - 0s 310us/step - loss: 0.1602 - accuracy: 0.9381 - val_loss: 0.1783 - val_accuracy: 0.9304
    Epoch 25/30
    1000/1000 [==============================] - 0s 316us/step - loss: 0.1503 - accuracy: 0.9426 - val_loss: 0.1863 - val_accuracy: 0.9250
    Epoch 26/30
    1000/1000 [==============================] - 0s 321us/step - loss: 0.1487 - accuracy: 0.9450 - val_loss: 0.1781 - val_accuracy: 0.9313
    Epoch 27/30
    1000/1000 [==============================] - 0s 315us/step - loss: 0.1516 - accuracy: 0.9402 - val_loss: 0.1790 - val_accuracy: 0.9301
    Epoch 28/30
    1000/1000 [==============================] - 0s 315us/step - loss: 0.1494 - accuracy: 0.9429 - val_loss: 0.1693 - val_accuracy: 0.9344
    Epoch 29/30
    1000/1000 [==============================] - 0s 317us/step - loss: 0.1460 - accuracy: 0.9455 - val_loss: 0.1616 - val_accuracy: 0.9386
    Epoch 30/30
    1000/1000 [==============================] - 0s 309us/step - loss: 0.1388 - accuracy: 0.9495 - val_loss: 0.1621 - val_accuracy: 0.9367



```python
plot_model_result(history,weights,final_w)
```

## 2(b). Neural Network with 2 Hidden Layer and 3 Hidden Layer 

## 2 Hidden Layer without Regularization


```python
model = Sequential()
model.add(Dense(30, activation='sigmoid', input_dim=xtrain.shape[1]))
model.add(Dense(30, activation='sigmoid'))
model.add(Dense(10, activation='softmax'))
weights1 = []
weights2 = []
save_weights1 = LambdaCallback(on_epoch_end=lambda batch, logs: weights1.append(model.layers[0].get_weights()[0]))
save_weights2 = LambdaCallback(on_epoch_end=lambda batch, logs: weights2.append(model.layers[1].get_weights()[0]))
sgd = optimizers.SGD(learning_rate=0.1)
model.compile(optimizer=sgd,
              loss='binary_crossentropy',
              metrics=['accuracy'])
history = model.fit(xtrain, label_train,validation_data=(xtest, label_test), 
                    epochs=30, batch_size=10, shuffle = False,callbacks = [save_weights1,save_weights2])
final_w = [model.layers[0].get_weights()[0],model.layers[1].get_weights()[0]]
weights = [weights1,weights2]
```

    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
    1000/1000 [==============================] - 1s 523us/step - loss: 0.3258 - accuracy: 0.9000 - val_loss: 0.3214 - val_accuracy: 0.9000
    Epoch 2/30
    1000/1000 [==============================] - 0s 327us/step - loss: 0.3157 - accuracy: 0.9000 - val_loss: 0.3153 - val_accuracy: 0.9000
    Epoch 3/30
    1000/1000 [==============================] - 0s 325us/step - loss: 0.3090 - accuracy: 0.9000 - val_loss: 0.3101 - val_accuracy: 0.9000
    Epoch 4/30
    1000/1000 [==============================] - 0s 326us/step - loss: 0.3043 - accuracy: 0.9000 - val_loss: 0.3056 - val_accuracy: 0.9000
    Epoch 5/30
    1000/1000 [==============================] - 0s 332us/step - loss: 0.2982 - accuracy: 0.9000 - val_loss: 0.3004 - val_accuracy: 0.9000
    Epoch 6/30
    1000/1000 [==============================] - 0s 328us/step - loss: 0.2922 - accuracy: 0.9000 - val_loss: 0.2953 - val_accuracy: 0.9000
    Epoch 7/30
    1000/1000 [==============================] - 0s 331us/step - loss: 0.2861 - accuracy: 0.9000 - val_loss: 0.2892 - val_accuracy: 0.9000
    Epoch 8/30
    1000/1000 [==============================] - 0s 328us/step - loss: 0.2786 - accuracy: 0.9000 - val_loss: 0.2817 - val_accuracy: 0.9000
    Epoch 9/30
    1000/1000 [==============================] - 0s 333us/step - loss: 0.2722 - accuracy: 0.9000 - val_loss: 0.2769 - val_accuracy: 0.9000
    Epoch 10/30
    1000/1000 [==============================] - 0s 330us/step - loss: 0.2658 - accuracy: 0.9000 - val_loss: 0.2700 - val_accuracy: 0.9000
    Epoch 11/30
    1000/1000 [==============================] - 0s 331us/step - loss: 0.2587 - accuracy: 0.9000 - val_loss: 0.2649 - val_accuracy: 0.9000
    Epoch 12/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.2528 - accuracy: 0.9000 - val_loss: 0.2602 - val_accuracy: 0.9000
    Epoch 13/30
    1000/1000 [==============================] - 0s 325us/step - loss: 0.2452 - accuracy: 0.9000 - val_loss: 0.2529 - val_accuracy: 0.9000
    Epoch 14/30
    1000/1000 [==============================] - 0s 316us/step - loss: 0.2384 - accuracy: 0.9000 - val_loss: 0.2474 - val_accuracy: 0.9000
    Epoch 15/30
    1000/1000 [==============================] - 0s 326us/step - loss: 0.2318 - accuracy: 0.9025 - val_loss: 0.2447 - val_accuracy: 0.9006
    Epoch 16/30
    1000/1000 [==============================] - 0s 337us/step - loss: 0.2252 - accuracy: 0.9059 - val_loss: 0.2338 - val_accuracy: 0.9059
    Epoch 17/30
    1000/1000 [==============================] - 0s 322us/step - loss: 0.2179 - accuracy: 0.9087 - val_loss: 0.2302 - val_accuracy: 0.9081
    Epoch 18/30
    1000/1000 [==============================] - 0s 329us/step - loss: 0.2111 - accuracy: 0.9138 - val_loss: 0.2264 - val_accuracy: 0.9102
    Epoch 19/30
    1000/1000 [==============================] - 0s 328us/step - loss: 0.2074 - accuracy: 0.9155 - val_loss: 0.2185 - val_accuracy: 0.9145
    Epoch 20/30
    1000/1000 [==============================] - 0s 326us/step - loss: 0.1993 - accuracy: 0.9179 - val_loss: 0.2126 - val_accuracy: 0.9175
    Epoch 21/30
    1000/1000 [==============================] - 0s 330us/step - loss: 0.1926 - accuracy: 0.9230 - val_loss: 0.2121 - val_accuracy: 0.9186
    Epoch 22/30
    1000/1000 [==============================] - 0s 330us/step - loss: 0.1946 - accuracy: 0.9208 - val_loss: 0.2116 - val_accuracy: 0.9168
    Epoch 23/30
    1000/1000 [==============================] - 0s 325us/step - loss: 0.1900 - accuracy: 0.9225 - val_loss: 0.2027 - val_accuracy: 0.9211
    Epoch 24/30
    1000/1000 [==============================] - 0s 321us/step - loss: 0.1810 - accuracy: 0.9269 - val_loss: 0.1967 - val_accuracy: 0.9244
    Epoch 25/30
    1000/1000 [==============================] - 0s 317us/step - loss: 0.1746 - accuracy: 0.9295 - val_loss: 0.1957 - val_accuracy: 0.9243
    Epoch 26/30
    1000/1000 [==============================] - 0s 325us/step - loss: 0.1716 - accuracy: 0.9282 - val_loss: 0.1906 - val_accuracy: 0.9255
    Epoch 27/30
    1000/1000 [==============================] - 0s 324us/step - loss: 0.1710 - accuracy: 0.9305 - val_loss: 0.1938 - val_accuracy: 0.9264
    Epoch 28/30
    1000/1000 [==============================] - 0s 324us/step - loss: 0.1661 - accuracy: 0.9332 - val_loss: 0.1813 - val_accuracy: 0.9291
    Epoch 29/30
    1000/1000 [==============================] - 0s 323us/step - loss: 0.1622 - accuracy: 0.9361 - val_loss: 0.1795 - val_accuracy: 0.9295
    Epoch 30/30
    1000/1000 [==============================] - 0s 326us/step - loss: 0.1555 - accuracy: 0.9374 - val_loss: 0.1821 - val_accuracy: 0.9289



```python
plot_model_result(history,weights,final_w)
```

## 3 Hidden Layer Without Regularization


```python
model = Sequential()
model.add(Dense(30, activation='sigmoid', input_dim=xtrain.shape[1]))
model.add(Dense(30, activation='sigmoid'))
model.add(Dense(30, activation='sigmoid'))
model.add(Dense(10, activation='softmax'))
weights1 = []
weights2 = []
weights3 = []
save_weights1 = LambdaCallback(on_epoch_end=lambda batch, logs: weights1.append(model.layers[0].get_weights()[0]))
save_weights2 = LambdaCallback(on_epoch_end=lambda batch, logs: weights2.append(model.layers[1].get_weights()[0]))
save_weights3 = LambdaCallback(on_epoch_end=lambda batch, logs: weights3.append(model.layers[1].get_weights()[0]))
sgd = optimizers.SGD(learning_rate=0.1)
model.compile(optimizer=sgd,
              loss='binary_crossentropy',
              metrics=['accuracy'])
history = model.fit(xtrain, label_train,validation_data=(xtest, label_test), 
                    epochs=30, batch_size=10, shuffle = False,callbacks = [save_weights1,save_weights2,save_weights3])
final_w = [model.layers[0].get_weights()[0],model.layers[1].get_weights()[0],model.layers[2].get_weights()[0]]
weights = [weights1,weights2,weights3]
```

    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
    1000/1000 [==============================] - 1s 561us/step - loss: 0.3428 - accuracy: 0.9000 - val_loss: 0.3306 - val_accuracy: 0.9000
    Epoch 2/30
    1000/1000 [==============================] - 0s 339us/step - loss: 0.3274 - accuracy: 0.9000 - val_loss: 0.3247 - val_accuracy: 0.9000
    Epoch 3/30
    1000/1000 [==============================] - 0s 335us/step - loss: 0.3242 - accuracy: 0.9000 - val_loss: 0.3234 - val_accuracy: 0.9000
    Epoch 4/30
    1000/1000 [==============================] - 0s 341us/step - loss: 0.3231 - accuracy: 0.9000 - val_loss: 0.3231 - val_accuracy: 0.9000
    Epoch 5/30
    1000/1000 [==============================] - 0s 346us/step - loss: 0.3226 - accuracy: 0.9000 - val_loss: 0.3227 - val_accuracy: 0.9000
    Epoch 6/30
    1000/1000 [==============================] - 0s 340us/step - loss: 0.3220 - accuracy: 0.9000 - val_loss: 0.3221 - val_accuracy: 0.9000
    Epoch 7/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.3215 - accuracy: 0.9000 - val_loss: 0.3217 - val_accuracy: 0.9000
    Epoch 8/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.3209 - accuracy: 0.9000 - val_loss: 0.3212 - val_accuracy: 0.9000
    Epoch 9/30
    1000/1000 [==============================] - 0s 335us/step - loss: 0.3203 - accuracy: 0.9000 - val_loss: 0.3208 - val_accuracy: 0.9000
    Epoch 10/30
    1000/1000 [==============================] - 0s 330us/step - loss: 0.3198 - accuracy: 0.9000 - val_loss: 0.3204 - val_accuracy: 0.9000
    Epoch 11/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.3193 - accuracy: 0.9000 - val_loss: 0.3199 - val_accuracy: 0.9000
    Epoch 12/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.3187 - accuracy: 0.9000 - val_loss: 0.3195 - val_accuracy: 0.9000
    Epoch 13/30
    1000/1000 [==============================] - 0s 332us/step - loss: 0.3182 - accuracy: 0.9000 - val_loss: 0.3189 - val_accuracy: 0.9000
    Epoch 14/30
    1000/1000 [==============================] - 0s 332us/step - loss: 0.3176 - accuracy: 0.9000 - val_loss: 0.3184 - val_accuracy: 0.9000
    Epoch 15/30
    1000/1000 [==============================] - 0s 337us/step - loss: 0.3169 - accuracy: 0.9000 - val_loss: 0.3178 - val_accuracy: 0.9000
    Epoch 16/30
    1000/1000 [==============================] - 0s 336us/step - loss: 0.3162 - accuracy: 0.9000 - val_loss: 0.3170 - val_accuracy: 0.9000
    Epoch 17/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.3153 - accuracy: 0.9000 - val_loss: 0.3165 - val_accuracy: 0.9000
    Epoch 18/30
    1000/1000 [==============================] - 0s 333us/step - loss: 0.3145 - accuracy: 0.9000 - val_loss: 0.3156 - val_accuracy: 0.9000
    Epoch 19/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.3136 - accuracy: 0.9000 - val_loss: 0.3148 - val_accuracy: 0.9000
    Epoch 20/30
    1000/1000 [==============================] - 0s 332us/step - loss: 0.3125 - accuracy: 0.9000 - val_loss: 0.3138 - val_accuracy: 0.9000
    Epoch 21/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.3115 - accuracy: 0.9000 - val_loss: 0.3129 - val_accuracy: 0.9000
    Epoch 22/30
    1000/1000 [==============================] - 0s 333us/step - loss: 0.3104 - accuracy: 0.9000 - val_loss: 0.3120 - val_accuracy: 0.9000
    Epoch 23/30
    1000/1000 [==============================] - 0s 336us/step - loss: 0.3093 - accuracy: 0.9000 - val_loss: 0.3106 - val_accuracy: 0.9000
    Epoch 24/30
    1000/1000 [==============================] - 0s 333us/step - loss: 0.3078 - accuracy: 0.9000 - val_loss: 0.3095 - val_accuracy: 0.9000
    Epoch 25/30
    1000/1000 [==============================] - 0s 335us/step - loss: 0.3064 - accuracy: 0.9000 - val_loss: 0.3082 - val_accuracy: 0.9000
    Epoch 26/30
    1000/1000 [==============================] - 0s 335us/step - loss: 0.3048 - accuracy: 0.9000 - val_loss: 0.3068 - val_accuracy: 0.9000
    Epoch 27/30
    1000/1000 [==============================] - 0s 336us/step - loss: 0.3033 - accuracy: 0.9000 - val_loss: 0.3049 - val_accuracy: 0.9000
    Epoch 28/30
    1000/1000 [==============================] - 0s 329us/step - loss: 0.3015 - accuracy: 0.9000 - val_loss: 0.3031 - val_accuracy: 0.9000
    Epoch 29/30
    1000/1000 [==============================] - 0s 336us/step - loss: 0.2994 - accuracy: 0.9000 - val_loss: 0.3012 - val_accuracy: 0.9000
    Epoch 30/30
    1000/1000 [==============================] - 0s 333us/step - loss: 0.2973 - accuracy: 0.9000 - val_loss: 0.2993 - val_accuracy: 0.9000



```python
plot_model_result(history,weights,final_w)
```

## 1 Hidden layer with Regularization - l2(5)


```python
model = Sequential()
model.add(Dense(30, activation='sigmoid', input_dim=xtrain.shape[1],
                kernel_regularizer=regularizers.l2(5),
                bias_regularizer=regularizers.l2(5)))
model.add(Dense(10, activation='softmax',
                kernel_regularizer=regularizers.l2(5),
                bias_regularizer=regularizers.l2(5)))
weights = []
save_weights = LambdaCallback(on_epoch_end=lambda batch, logs: weights.append(model.layers[0].get_weights()[0]))
sgd = optimizers.SGD(learning_rate=0.1)
model.compile(optimizer=sgd,
              loss='binary_crossentropy',
              metrics=['accuracy'])
history = model.fit(xtrain, label_train,validation_data=(xtest, label_test), 
                    epochs=30, batch_size=10, shuffle = False,callbacks = [save_weights])
final_w = [model.layers[0].get_weights()[0]]
weights = [weights]
```

    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
    1000/1000 [==============================] - 1s 567us/step - loss: 3.9812 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 2/30
    1000/1000 [==============================] - 0s 322us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 3/30
    1000/1000 [==============================] - 0s 321us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 4/30
    1000/1000 [==============================] - 0s 322us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 5/30
    1000/1000 [==============================] - 0s 323us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 6/30
    1000/1000 [==============================] - 0s 324us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 7/30
    1000/1000 [==============================] - 0s 324us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 8/30
    1000/1000 [==============================] - 0s 318us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 9/30
    1000/1000 [==============================] - 0s 321us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 10/30
    1000/1000 [==============================] - 0s 318us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 11/30
    1000/1000 [==============================] - 0s 319us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 12/30
    1000/1000 [==============================] - 0s 322us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 13/30
    1000/1000 [==============================] - 0s 319us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 14/30
    1000/1000 [==============================] - 0s 322us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 15/30
    1000/1000 [==============================] - 0s 319us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 16/30
    1000/1000 [==============================] - 0s 320us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 17/30
    1000/1000 [==============================] - 0s 317us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 18/30
    1000/1000 [==============================] - 0s 314us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 19/30
    1000/1000 [==============================] - 0s 316us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 20/30
    1000/1000 [==============================] - 0s 316us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 21/30
    1000/1000 [==============================] - 0s 319us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 22/30
    1000/1000 [==============================] - 0s 312us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 23/30
    1000/1000 [==============================] - 0s 317us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 24/30
    1000/1000 [==============================] - 0s 315us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 25/30
    1000/1000 [==============================] - 0s 320us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 26/30
    1000/1000 [==============================] - 0s 320us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 27/30
    1000/1000 [==============================] - 0s 320us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 28/30
    1000/1000 [==============================] - 0s 321us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 29/30
    1000/1000 [==============================] - 0s 317us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 30/30
    1000/1000 [==============================] - 0s 317us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000



```python
plot_model_result(history,weights,final_w)
```

## 2 Hidden Layer with Regularization - l2(5)


```python
model = Sequential()
model.add(Dense(30, activation='sigmoid', input_dim=xtrain.shape[1],
                kernel_regularizer=regularizers.l2(5),
                bias_regularizer=regularizers.l2(5)))
model.add(Dense(30, activation='sigmoid',
                kernel_regularizer=regularizers.l2(5),
                bias_regularizer=regularizers.l2(5)))
model.add(Dense(10, activation='softmax',
                kernel_regularizer=regularizers.l2(5),
                bias_regularizer=regularizers.l2(5)))
weights1 = []
weights2 = []
save_weights1 = LambdaCallback(on_epoch_end=lambda batch, logs: weights1.append(model.layers[0].get_weights()[0]))
save_weights2 = LambdaCallback(on_epoch_end=lambda batch, logs: weights2.append(model.layers[1].get_weights()[0]))
sgd = optimizers.SGD(learning_rate=0.1)
model.compile(optimizer=sgd,
              loss='binary_crossentropy',
              metrics=['accuracy'])
history = model.fit(xtrain, label_train,validation_data=(xtest, label_test), 
                    epochs=30, batch_size=10, shuffle = False,callbacks = [save_weights1,save_weights2])
final_w = [model.layers[0].get_weights()[0],model.layers[1].get_weights()[0]]
weights = [weights1,weights2]
```

    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
    1000/1000 [==============================] - 1s 630us/step - loss: 5.4465 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 2/30
    1000/1000 [==============================] - 0s 329us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 3/30
    1000/1000 [==============================] - 0s 328us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 4/30
    1000/1000 [==============================] - 0s 326us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 5/30
    1000/1000 [==============================] - 0s 328us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 6/30
    1000/1000 [==============================] - 0s 330us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 7/30
    1000/1000 [==============================] - 0s 332us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 8/30
    1000/1000 [==============================] - 0s 324us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 9/30
    1000/1000 [==============================] - 0s 328us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 10/30
    1000/1000 [==============================] - 0s 333us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 11/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 12/30
    1000/1000 [==============================] - 0s 330us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 13/30
    1000/1000 [==============================] - 0s 328us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 14/30
    1000/1000 [==============================] - 0s 330us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 15/30
    1000/1000 [==============================] - 0s 330us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 16/30
    1000/1000 [==============================] - 0s 331us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 17/30
    1000/1000 [==============================] - 0s 328us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 18/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 19/30
    1000/1000 [==============================] - 0s 331us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 20/30
    1000/1000 [==============================] - 0s 325us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 21/30
    1000/1000 [==============================] - 0s 329us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 22/30
    1000/1000 [==============================] - 0s 336us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 23/30
    1000/1000 [==============================] - 0s 331us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 24/30
    1000/1000 [==============================] - 0s 331us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 25/30
    1000/1000 [==============================] - 0s 331us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 26/30
    1000/1000 [==============================] - 0s 333us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 27/30
    1000/1000 [==============================] - 0s 335us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 28/30
    1000/1000 [==============================] - 0s 336us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 29/30
    1000/1000 [==============================] - 0s 330us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 30/30
    1000/1000 [==============================] - 0s 333us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000



```python
plot_model_result(history,weights,final_w)
```

## 3 Hidden Layer with Regularization - l2(5)


```python
model = Sequential()
model.add(Dense(30, activation='sigmoid', input_dim=xtrain.shape[1],
                kernel_regularizer=regularizers.l2(5),
                bias_regularizer=regularizers.l2(5)))
model.add(Dense(30, activation='sigmoid',
                kernel_regularizer=regularizers.l2(5),
                bias_regularizer=regularizers.l2(5)))
model.add(Dense(30, activation='sigmoid',
                kernel_regularizer=regularizers.l2(5),
                bias_regularizer=regularizers.l2(5)))
model.add(Dense(10, activation='softmax',
                kernel_regularizer=regularizers.l2(5),
                bias_regularizer=regularizers.l2(5)))
weights1 = []
weights2 = []
weights3 = []
save_weights1 = LambdaCallback(on_epoch_end=lambda batch, logs: weights1.append(model.layers[0].get_weights()[0]))
save_weights2 = LambdaCallback(on_epoch_end=lambda batch, logs: weights2.append(model.layers[1].get_weights()[0]))
save_weights3 = LambdaCallback(on_epoch_end=lambda batch, logs: weights3.append(model.layers[1].get_weights()[0]))
sgd = optimizers.SGD(learning_rate=0.1)
model.compile(optimizer=sgd,
              loss='binary_crossentropy',
              metrics=['accuracy'])
history = model.fit(xtrain, label_train,validation_data=(xtest, label_test), 
                    epochs=30, batch_size=10, shuffle = False,callbacks = [save_weights1,save_weights2,save_weights3])
final_w = [model.layers[0].get_weights()[0],model.layers[1].get_weights()[0],model.layers[2].get_weights()[0]]
weights = [weights1,weights2,weights3]
```

    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
    1000/1000 [==============================] - 1s 695us/step - loss: 6.9392 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 2/30
    1000/1000 [==============================] - 0s 355us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 3/30
    1000/1000 [==============================] - 0s 355us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 4/30
    1000/1000 [==============================] - 0s 356us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 5/30
    1000/1000 [==============================] - 0s 357us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 6/30
    1000/1000 [==============================] - 0s 352us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 7/30
    1000/1000 [==============================] - 0s 354us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 8/30
    1000/1000 [==============================] - 0s 352us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 9/30
    1000/1000 [==============================] - 0s 348us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 10/30
    1000/1000 [==============================] - 0s 350us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 11/30
    1000/1000 [==============================] - 0s 351us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 12/30
    1000/1000 [==============================] - 0s 351us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 13/30
    1000/1000 [==============================] - 0s 352us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 14/30
    1000/1000 [==============================] - 0s 350us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 15/30
    1000/1000 [==============================] - 0s 350us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 16/30
    1000/1000 [==============================] - 0s 350us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 17/30
    1000/1000 [==============================] - 0s 351us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 18/30
    1000/1000 [==============================] - 0s 354us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 19/30
    1000/1000 [==============================] - 0s 350us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 20/30
    1000/1000 [==============================] - 0s 349us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 21/30
    1000/1000 [==============================] - 0s 347us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 22/30
    1000/1000 [==============================] - 0s 349us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 23/30
    1000/1000 [==============================] - 0s 348us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 24/30
    1000/1000 [==============================] - 0s 353us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 25/30
    1000/1000 [==============================] - 0s 351us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 26/30
    1000/1000 [==============================] - 0s 357us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 27/30
    1000/1000 [==============================] - 0s 353us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 28/30
    1000/1000 [==============================] - 0s 356us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 29/30
    1000/1000 [==============================] - 0s 351us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 30/30
    1000/1000 [==============================] - 0s 347us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000



```python
plot_model_result(history,weights,final_w)
```

## 2(c) . Construct and Train Convolutional Neural network
For Convolutional Neural Networks we first change shape of th Datasets from set of array of 1 Dimensional array of $28x28$ image to a set of 3-Dimensinal Image with Value.


```python
x_train_imgs = xtrain.reshape(xtrain.shape[0],28,28,1)
x_test_imgs = xtrain.reshape(xtest.shape[0],28,28,1)
```

### Preprocessing the Image :
We use th ImageDataGenerator library from keras to generate a augumented form of out original training dataset.


```python
from keras.preprocessing.image import ImageDataGenerator
aug_data = ImageDataGenerator(rotation_range=3,
                              width_shift_range=0.25, # 0.25 * 28 = 8 pixels approx
                              height_shift_range=0.25, # 0.25 * 28 = 8 pixels approx
                              horizontal_flip=True,
                              fill_mode="nearest")
aug_data.fit(x_train_imgs)
n = 0
for x_batch, y_batch in aug_data.flow(x_train_imgs, label_train, batch_size=1000):
    x_batch = np.reshape(x_batch, [-1, 28*28])
    model.fit(x_batch, y_batch)
    n += 1
    if n >= len(x_train_imgs) / 1000:
        break
x_batch = x_batch.reshape(x_batch.shape[0],28,28,1)               
```

    Epoch 1/1
    1000/1000 [==============================] - 0s 78us/step - loss: 0.3252 - accuracy: 0.9000


Now we build the Convolutional Neutral Network and feed the Augumented Training Dataset to our model.


```python
model = Sequential()
model.add(Conv2D(30, activation='sigmoid',kernel_size=(5, 5), input_shape=(28,28,1)))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
model.add(Flatten())
model.add(Dropout(0.25))
model.add(Dense(10, activation='softmax'))
weights = []
save_weights = LambdaCallback(on_epoch_end=lambda batch, logs: weights.append(model.layers[0].get_weights()[0]))
sgd = optimizers.SGD(learning_rate=0.1)
model.compile(optimizer=sgd,
              loss='binary_crossentropy',
              metrics=['accuracy'])
history = model.fit(x_batch, y_batch,validation_data=(x_test_imgs, label_test), 
                    epochs=30, batch_size=10, shuffle = False,callbacks = [save_weights])
final_w = [model.layers[0].get_weights()[0]]
weights = [weights]
```

    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
    1000/1000 [==============================] - 1s 1ms/step - loss: 0.7134 - accuracy: 0.8686 - val_loss: 0.3678 - val_accuracy: 0.8987
    Epoch 2/30
    1000/1000 [==============================] - 1s 883us/step - loss: 0.3261 - accuracy: 0.8984 - val_loss: 0.3791 - val_accuracy: 0.8962
    Epoch 3/30
    1000/1000 [==============================] - 1s 874us/step - loss: 0.3104 - accuracy: 0.8997 - val_loss: 0.3911 - val_accuracy: 0.8927
    Epoch 4/30
    1000/1000 [==============================] - 1s 881us/step - loss: 0.2959 - accuracy: 0.9031 - val_loss: 0.3962 - val_accuracy: 0.8924
    Epoch 5/30
    1000/1000 [==============================] - 1s 880us/step - loss: 0.2871 - accuracy: 0.9046 - val_loss: 0.4069 - val_accuracy: 0.8873
    Epoch 6/30
    1000/1000 [==============================] - 1s 878us/step - loss: 0.2766 - accuracy: 0.9055 - val_loss: 0.4134 - val_accuracy: 0.8900
    Epoch 7/30
    1000/1000 [==============================] - 1s 871us/step - loss: 0.2653 - accuracy: 0.9073 - val_loss: 0.4232 - val_accuracy: 0.8732
    Epoch 8/30
    1000/1000 [==============================] - 1s 880us/step - loss: 0.2541 - accuracy: 0.9090 - val_loss: 0.4219 - val_accuracy: 0.8821
    Epoch 9/30
    1000/1000 [==============================] - 1s 880us/step - loss: 0.2451 - accuracy: 0.9130 - val_loss: 0.4370 - val_accuracy: 0.8836
    Epoch 10/30
    1000/1000 [==============================] - 1s 884us/step - loss: 0.2386 - accuracy: 0.9130 - val_loss: 0.4234 - val_accuracy: 0.8808
    Epoch 11/30
    1000/1000 [==============================] - 1s 891us/step - loss: 0.2294 - accuracy: 0.9174 - val_loss: 0.4404 - val_accuracy: 0.8792
    Epoch 12/30
    1000/1000 [==============================] - 1s 887us/step - loss: 0.2202 - accuracy: 0.9187 - val_loss: 0.4411 - val_accuracy: 0.8747
    Epoch 13/30
    1000/1000 [==============================] - 1s 879us/step - loss: 0.2136 - accuracy: 0.9212 - val_loss: 0.4486 - val_accuracy: 0.8742
    Epoch 14/30
    1000/1000 [==============================] - 1s 884us/step - loss: 0.2072 - accuracy: 0.9237 - val_loss: 0.4560 - val_accuracy: 0.8691
    Epoch 15/30
    1000/1000 [==============================] - 1s 888us/step - loss: 0.2011 - accuracy: 0.9252 - val_loss: 0.4632 - val_accuracy: 0.8716
    Epoch 16/30
    1000/1000 [==============================] - 1s 882us/step - loss: 0.1938 - accuracy: 0.9269 - val_loss: 0.4816 - val_accuracy: 0.8676
    Epoch 17/30
    1000/1000 [==============================] - 1s 879us/step - loss: 0.1862 - accuracy: 0.9307 - val_loss: 0.4618 - val_accuracy: 0.8724
    Epoch 18/30
    1000/1000 [==============================] - 1s 869us/step - loss: 0.1757 - accuracy: 0.9356 - val_loss: 0.4740 - val_accuracy: 0.8679
    Epoch 19/30
    1000/1000 [==============================] - 1s 874us/step - loss: 0.1700 - accuracy: 0.9373 - val_loss: 0.5001 - val_accuracy: 0.8602
    Epoch 20/30
    1000/1000 [==============================] - 1s 867us/step - loss: 0.1668 - accuracy: 0.9384 - val_loss: 0.4896 - val_accuracy: 0.8624
    Epoch 21/30
    1000/1000 [==============================] - 1s 898us/step - loss: 0.1617 - accuracy: 0.9416 - val_loss: 0.4837 - val_accuracy: 0.8652
    Epoch 22/30
    1000/1000 [==============================] - 1s 877us/step - loss: 0.1561 - accuracy: 0.9414 - val_loss: 0.5013 - val_accuracy: 0.8626
    Epoch 23/30
    1000/1000 [==============================] - 1s 872us/step - loss: 0.1498 - accuracy: 0.9442 - val_loss: 0.5084 - val_accuracy: 0.8612
    Epoch 24/30
    1000/1000 [==============================] - 1s 866us/step - loss: 0.1417 - accuracy: 0.9490 - val_loss: 0.5055 - val_accuracy: 0.8630
    Epoch 25/30
    1000/1000 [==============================] - 1s 870us/step - loss: 0.1356 - accuracy: 0.9500 - val_loss: 0.5158 - val_accuracy: 0.8606
    Epoch 26/30
    1000/1000 [==============================] - 1s 872us/step - loss: 0.1332 - accuracy: 0.9516 - val_loss: 0.5064 - val_accuracy: 0.8595
    Epoch 27/30
    1000/1000 [==============================] - 1s 872us/step - loss: 0.1319 - accuracy: 0.9520 - val_loss: 0.5240 - val_accuracy: 0.8585
    Epoch 28/30
    1000/1000 [==============================] - 1s 873us/step - loss: 0.1241 - accuracy: 0.9568 - val_loss: 0.5211 - val_accuracy: 0.8576
    Epoch 29/30
    1000/1000 [==============================] - 1s 871us/step - loss: 0.1264 - accuracy: 0.9538 - val_loss: 0.5257 - val_accuracy: 0.8551
    Epoch 30/30
    1000/1000 [==============================] - 1s 872us/step - loss: 0.1169 - accuracy: 0.9588 - val_loss: 0.5339 - val_accuracy: 0.8546



```python
plot_model_result(history,weights,final_w)
```


```python

```

7ff2d30575bda08fdd4a8ad13998db20a756e3b1
ipynb
Jupyter Notebook
MNIST_neuralnetworks.ipynb
fazildgr8/pattern_recognition
4a1636f950458ff57fd413c070758c2288157310
MIT
2020-07-02T23:40:16.000Z
2020-07-02T23:40:16.000Z
MNIST_neuralnetworks.ipynb
fazildgr8/pattern_recognition
4a1636f950458ff57fd413c070758c2288157310
MIT
MNIST_neuralnetworks.ipynb
fazildgr8/pattern_recognition
4a1636f950458ff57fd413c070758c2288157310
MIT
Qwen/Qwen-72B
1. YES
2. YES