<gh_stars>0
# -*- coding: utf-8 -*-
"""Deep_Learning_with_CNNs-1.ipynb

Automatically generated by Colaboratory.

Original file is located at
    https://colab.research.google.com/drive/1_-DhPtcW-mhY1iI_T3vDDI2PMIKwx55H

<a id="section-one"></a>

# **1. Abstract**

The dataset is composed of 16800 Arabic characters written by 60 participants in the age range of 19-40.

We choose to build it with keras API (Tensorflow backend) which is very intuitive. Firstly, we will prepare the data (handwritten digits images) then we will focus on the CNN modeling and evaluation.

We achieved 99.671% of accuracy with CNN trained model.

**For computational reasons, we set the number of steps (epochs) to 2.**

This Notebook follows three main parts:

* The data preparation
* The CNN modeling and evaluation
* The results prediction and submission

# **2 Preparations**
Preparing packages and data that will be used in the analysis process. Packages that will be loaded are mainly for data manipulation, data visualization and modeling. There are 2 datasets that are used in the analysis, they are train and test dataset. The main use of train dataset is to train models and use it to predict test dataset. While sample submission file is used to informed participants on the expected submission for the competition.
"""

!pip install --upgrade --force-reinstall --no-deps kaggle

!mkdir .kaggle

import json

token = {"username":"shubhangshahneu","key":"<KEY>"}

with open('/content/.kaggle/kaggle.json', 'w') as file:
    json.dump(token, file)

!mkdir ~/.kaggle

!cp /content/.kaggle/kaggle.json ~/.kaggle/kaggle.json && chmod 600 /root/.kaggle/kaggle.json

!kaggle config set -n path -v{/content}

!kaggle datasets download -d mloey1/ahcd1 -p /content

!unzip \*.zip

"""# **2.1. Importing necessary Libraries**"""

# Import the necessary libs
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import cv2
import seaborn as sn

import keras
from keras.models import Model
from keras.layers import Dense, Conv2D, MaxPooling2D, Flatten, Dropout, Input, BatchNormalization
# from keras.layers.normalization import BatchNormalization
from keras.callbacks import ModelCheckpoint

from keras.utils.np_utils import to_categorical
from keras.preprocessing.image import ImageDataGenerator
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix

arabic_characters = ['alef', 'beh', 'teh', 'theh', 'jeem', 'hah', 'khah', 'dal', 'thal',
                    'reh', 'zain', 'seen', 'sheen', 'sad', 'dad', 'tah', 'zah', 'ain',
                    'ghain', 'feh', 'qaf', 'kaf', 'lam', 'meem', 'noon', 'heh', 'waw', 'yeh']

"""<a id="section-two"></a>
# **2. Data preparation**

<a id="section-two-a"></a>
## 2.1 Load data
"""

# Commented out IPython magic to ensure Python compatibility.
# %ls

# loading the dataset
x_train = pd.read_csv("/content/csvTrainImages 13440x1024.csv",header=None).to_numpy()
y_train = pd.read_csv("/content/csvTrainLabel 13440x1.csv",header=None).to_numpy()-1 

x_test = pd.read_csv("/content/csvTestImages 3360x1024.csv",header=None).to_numpy()
y_test = pd.read_csv("/content/csvTestLabel 3360x1.csv",header=None).to_numpy()-1

print("x_train.shape =", x_train.shape, "\ny_train.shape =", y_train.shape, "\nx_test.shape =", x_test.shape, "\ny_test.shape =", y_test.shape)

"""Dataset has the following features:

Dataset size 13440 samples of handwritten images.
The size of each image is 28x28 pixels.

Each image has only 1 color channel, i.e., grayscale image.
Each pixel has value in the range of [0,255] where 0 represents black, and 255 represents white.

<a id="section-two-c"></a>
## 2.2 Normalization

We perform a grayscale normalization to reduce the effect of illumination's differences. 

Moreover the CNN converg faster on [0..1] data than on [0..255].
"""

# Normalize the data to make CNN faster
x_train = x_train / 255.0
x_test = x_test / 255.0

print(x_train.shape)
print(y_train.shape)

"""<a id="section-two-d"></a>
## 2.3 Reshape
"""

# Reshape image is 3D array (height = 28px, width = 28px , canal = 1)
x_train = x_train.reshape(-1,32,32,1)
x_test = x_test.reshape(-1,32,32,1)

print(x_train.shape)
print(y_train.shape)

print(x_test.shape)
print(y_test.shape)

"""Train and test images (32px x 32px) has been stock into pandas. We reshape all data to 32x32x1 3D matrices. 

Keras requires an extra dimension in the end which correspond to channels. 
"""

ra = np.random.randint(0, 13440, size=25)
plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.imshow(x_train[ra[i]].reshape(32,32).T,"gray")
    plt.xlabel(arabic_characters[int(y_train[ra[i]][0])])
plt.show()

"""<a id="section-two-e"></a>
## 2.5 Label encoding
"""

# Converting the class vector in integers to binary class matrix
y_train = to_categorical(y_train)
y_test = to_categorical(y_test)
print(y_train.shape, y_test.shape)
for i in zip(y_train[0], arabic_characters):
    print(i)

"""Labels are 28 characters of arabic_characters."""

print(y_train.shape)
print(y_test.shape)

"""# **PART A - DEEP LEARNING MODEL**

<a id="section-three"></a>
# 3. CNN

<a id="section-three-a"></a>
## 3.1 Define the model
"""

#Creating CNN model
def get_model(activationFn, lossFn, optimizer):
    In = Input(shape=(32,32,1))
    x = Conv2D(32, (5,5), padding="same", activation=activationFn)(In)
    x = Conv2D(32, (5,5), activation=activationFn)(x)
    x = Conv2D(32, (5,5), activation=activationFn)(x)
    x = MaxPooling2D((2,2))(x)
    x = BatchNormalization()(x)
    
    x = Conv2D(64, (5,5), padding="same", activation=activationFn)(x)
    x = Conv2D(64, (5,5), activation=activationFn)(x)
    x = Conv2D(64, (5,5), activation=activationFn)(x)
    x = MaxPooling2D((2,2))(x)
    x = BatchNormalization()(x)
    
    x = Flatten()(x)
    x = Dense(128, activation=activationFn)(x)
    x = Dense(128, activation=activationFn)(x)
    x = Dropout(0.4)(x)
    
    Out = Dense(28, activation="softmax")(x)
    
    model = Model(In, Out)
    model.compile(loss=lossFn, optimizer=optimizer, metrics=["accuracy"])
    return model

model = get_model("relu","categorical_crossentropy","adam")
keras.utils.vis_utils.plot_model(model, show_shapes=True, show_layer_names=True)

"""<a id="section-three-a"></a>
## 3.2 Data Augementation
"""

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 20

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

"""For the data augmentation, we chose to :
   - Randomly rotate some training images by 10 degrees
   - Randomly  Zoom by 10% some training images
   - Randomly shift images horizontally by 10% of the width
   - Randomly shift images vertically by 10% of the height
   
We did not apply a vertical_flip nor horizontal_flip since it could have lead to misclassify symetrical numbers such as 6 and 9.

Once our model is ready, we fit the training dataset .
"""

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

"""<a id="section-four"></a>
# 3.3. Evaluate the model

"""

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = model.predict(x_test)

# Evaluate model
score = model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""# **PART B - ACTIVATION FUNCTION**

We will now change the activation function **from reLU (Rectified Linear Unit) to seLU (Scaled Exponential Linear Unit)** and further observe, model efficiency.
"""

model = get_model("selu","categorical_crossentropy","adam")
keras.utils.vis_utils.plot_model(model, show_shapes=True, show_layer_names=True)

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 20

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = model.predict(x_test)

# Evaluate model
score = model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""After changing our activation function, the accuracy was decreased. Still, to further conclude the same, we will use another activation function.

We will now change our activation function from seLU to **Exponential linear unit (ELU)**.
"""

model = get_model("elu","categorical_crossentropy","adam")
keras.utils.vis_utils.plot_model(model, show_shapes=True, show_layer_names=True)

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 20

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = model.predict(x_test)

# Evaluate model
score = model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""After using reLU, seLU and ELU as our activation functions, its been observed that, **reLU** has the highest test accuracy as compared to others.

# **PART C - COST FUNCTION**

We will now change the activation function **from categorical_crossentropy to mean_squared_error** and further observe, model efficiency.
"""

model = get_model("relu","mean_squared_error","adam")
keras.utils.vis_utils.plot_model(model, show_shapes=True, show_layer_names=True)

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 20

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = model.predict(x_test)

# Evaluate model
score = model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""So, we can obseve that, after changing the cost function from cross-entropy to quadratic cost, our model's test accuracy was **decreased**.

Let's use another cost function - **hinge** to observe, how our model's accuracy is affected.
"""

model = get_model("relu","hinge","adam")
keras.utils.vis_utils.plot_model(model, show_shapes=True, show_layer_names=True)

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 20

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = model.predict(x_test)

# Evaluate model
score = model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""So, we can obseve that, after using cross-entropy, quadratic cost and hinge cost function, our model's test accuracy was best with **cross-entropy**.

# **PART D - EPOCHS**

Now, we have been using Epoch = 20, now we are going to first **double** the no. of epochs and then **reduce** the same by half, in order to observe, how our model's test accuracy changes.
"""

model = get_model("relu","categorical_crossentropy","adam")
keras.utils.vis_utils.plot_model(model, show_shapes=True, show_layer_names=True)

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 40

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = model.predict(x_test)

# Evaluate model
score = model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""We can observe that, our test accuracy was **increased** after doubling the no. of epochs from 20 to 40.

Now, lets see how the accuracy affects after changing the no of epochs to half, i.e 10.
"""

model = get_model("relu","categorical_crossentropy","adam")
keras.utils.vis_utils.plot_model(model, show_shapes=True, show_layer_names=True)

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 10

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = model.predict(x_test)

# Evaluate model
score = model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""So, we can see that, after reducing the no. of epochs, the test accuracy was also **decreased**.

# **PART E - GRADIENT ESTIMATION**

Now, we have been using 'adam' as our gradiant estimation. Now, lets change the gradiant estimation to '**rmsprop**' and further observe, model effciency.
"""

model = get_model("relu","categorical_crossentropy","rmsprop")
keras.utils.vis_utils.plot_model(model, show_shapes=True, show_layer_names=True)

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 40

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = model.predict(x_test)

# Evaluate model
score = model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""So, we can observe that, after using RMSprop as the gradiant estimator, test accuracy was **decreased**.

# **PART F - NETWORK ARCHITECTURE**

We will be reducing the no. of layers and further observe, how the test accuracy changes.
"""

#Creating CNN model
def get_network_model(activationFn, lossFn, optimizer):
    In = Input(shape=(32,32,1))
    x = Conv2D(32, (5,5), padding="same", activation=activationFn)(In)
    x = Conv2D(32, (5,5), activation=activationFn)(x)
    x = MaxPooling2D((2,2))(x)
    x = BatchNormalization()(x)
    
    x = Conv2D(64, (5,5), padding="same", activation=activationFn)(x)
    x = Conv2D(64, (5,5), activation=activationFn)(x)
    x = MaxPooling2D((2,2))(x)
    x = BatchNormalization()(x)
    
    x = Flatten()(x)
    x = Dense(128, activation=activationFn)(x)
    x = Dense(128, activation=activationFn)(x)
    x = Dropout(0.4)(x)
    
    Out = Dense(28, activation="softmax")(x)
    
    model = Model(In, Out)
    model.compile(loss=lossFn, optimizer=optimizer, metrics=["accuracy"])
    return model

network_model = get_network_model("relu","categorical_crossentropy","adam")
keras.utils.vis_utils.plot_model(network_model, show_shapes=True, show_layer_names=True)

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 40

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = network_model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = model.predict(x_test)

# Evaluate model
score = model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""So after reducing the no of layers, test accuracy was **decreased**.

# **PART G - NETWORK INITIALIZATION**

We will now change the network initialization to Glorot normal initializer, also called **Xavier** normal initializer and observe the test accuracy.
"""

#Creating CNN model
def get_network_initialization_model(activationFn, lossFn, optimizer):
    In = Input(shape=(32,32,1))
    x = Conv2D(32, (5,5), padding="same", activation=activationFn)(In)
    x = Conv2D(32, (5,5), activation=activationFn)(x)
    x = MaxPooling2D((2,2))(x)
    x = BatchNormalization()(x)
    
    x = Conv2D(64, (5,5), padding="same", activation=activationFn)(x)
    x = Conv2D(64, (5,5), activation=activationFn)(x)
    x = MaxPooling2D((2,2))(x)
    x = BatchNormalization()(x)
    
    x = Flatten()(x)
    x = Dense(128, activation=activationFn, kernel_initializer="glorot_normal")(x)
    x = Dense(128, activation=activationFn)(x)
    x = Dropout(0.4)(x)
    
    Out = Dense(28, activation="softmax")(x)
    
    model = Model(In, Out)
    model.compile(loss=lossFn, optimizer=optimizer, metrics=["accuracy"])
    return model

network_intialization_model = get_network_initialization_model("relu","categorical_crossentropy","adam")
keras.utils.vis_utils.plot_model(network_intialization_model, show_shapes=True, show_layer_names=True)

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 40

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = network_intialization_model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = network_intialization_model.predict(x_test)

# Evaluate model
score = network_intialization_model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""So, we can observe that, after changing the network initilizer to **Xavier**, our model's test accuracy was **increased**.

# **4. Conclusion**

So, after running our model through various scenarios, we can conclude that:

1. Activation functions, do affect the overall model's accuracy. As in our case, after using relu, selu and elu, we found that, **relu** had the best test accuracy.

2. Cost function also affects the test accuracy and after using Quadratic, cross-entropy and hinge, we found that, **cross-entropy** was best suited for our model. 

3. No. of epochs also affects the test accuracy and in our model, after **doubling** the no. of epochs, test accuracy was increased and reducing the no. of epochs, further decreased accuracy.

4. After changing the gradiant estimators, we found that with **adam**, our overall test accuracy was increased. Thus, we can say that, gradiant estimator also plays a significant role in model's performance.

5. We changed the no. of layers, and then concluded that, after decreasing the no. of layers, our model's test accuracy was also decreased.

6. After changing the network initilizer to **Xavier**, our model's test accuracy was increased. So, we can conclude that, network initializaion is also important for model's accuracy.

# **5. Refrence**

https://www.kaggle.com/mloey1/ahcd1/

https://www.kaggle.com/yehyachali/arabic-mnist-with-detection/notebook

https://keras.io/about/\n

Copyright 2021 <NAME>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
"""
# -*- coding: utf-8 -*-
"""Deep_Learning_with_CNNs-1.ipynb

Automatically generated by Colaboratory.

Original file is located at
    https://colab.research.google.com/drive/1_-DhPtcW-mhY1iI_T3vDDI2PMIKwx55H

<a id="section-one"></a>

# **1. Abstract**

The dataset is composed of 16800 Arabic characters written by 60 participants in the age range of 19-40.

We choose to build it with keras API (Tensorflow backend) which is very intuitive. Firstly, we will prepare the data (handwritten digits images) then we will focus on the CNN modeling and evaluation.

We achieved 99.671% of accuracy with CNN trained model.

**For computational reasons, we set the number of steps (epochs) to 2.**

This Notebook follows three main parts:

* The data preparation
* The CNN modeling and evaluation
* The results prediction and submission

# **2 Preparations**
Preparing packages and data that will be used in the analysis process. Packages that will be loaded are mainly for data manipulation, data visualization and modeling. There are 2 datasets that are used in the analysis, they are train and test dataset. The main use of train dataset is to train models and use it to predict test dataset. While sample submission file is used to informed participants on the expected submission for the competition.
"""

!pip install --upgrade --force-reinstall --no-deps kaggle

!mkdir .kaggle

import json

token = {"username":"shubhangshahneu","key":"<KEY>"}

with open('/content/.kaggle/kaggle.json', 'w') as file:
    json.dump(token, file)

!mkdir ~/.kaggle

!cp /content/.kaggle/kaggle.json ~/.kaggle/kaggle.json && chmod 600 /root/.kaggle/kaggle.json

!kaggle config set -n path -v{/content}

!kaggle datasets download -d mloey1/ahcd1 -p /content

!unzip \*.zip

"""# **2.1. Importing necessary Libraries**"""

# Import the necessary libs
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import cv2
import seaborn as sn

import keras
from keras.models import Model
from keras.layers import Dense, Conv2D, MaxPooling2D, Flatten, Dropout, Input, BatchNormalization
# from keras.layers.normalization import BatchNormalization
from keras.callbacks import ModelCheckpoint

from keras.utils.np_utils import to_categorical
from keras.preprocessing.image import ImageDataGenerator
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix

arabic_characters = ['alef', 'beh', 'teh', 'theh', 'jeem', 'hah', 'khah', 'dal', 'thal',
                    'reh', 'zain', 'seen', 'sheen', 'sad', 'dad', 'tah', 'zah', 'ain',
                    'ghain', 'feh', 'qaf', 'kaf', 'lam', 'meem', 'noon', 'heh', 'waw', 'yeh']

"""<a id="section-two"></a>
# **2. Data preparation**

<a id="section-two-a"></a>
## 2.1 Load data
"""

# Commented out IPython magic to ensure Python compatibility.
# %ls

# loading the dataset
x_train = pd.read_csv("/content/csvTrainImages 13440x1024.csv",header=None).to_numpy()
y_train = pd.read_csv("/content/csvTrainLabel 13440x1.csv",header=None).to_numpy()-1 

x_test = pd.read_csv("/content/csvTestImages 3360x1024.csv",header=None).to_numpy()
y_test = pd.read_csv("/content/csvTestLabel 3360x1.csv",header=None).to_numpy()-1

print("x_train.shape =", x_train.shape, "\ny_train.shape =", y_train.shape, "\nx_test.shape =", x_test.shape, "\ny_test.shape =", y_test.shape)

"""Dataset has the following features:

Dataset size 13440 samples of handwritten images.
The size of each image is 28x28 pixels.

Each image has only 1 color channel, i.e., grayscale image.
Each pixel has value in the range of [0,255] where 0 represents black, and 255 represents white.

<a id="section-two-c"></a>
## 2.2 Normalization

We perform a grayscale normalization to reduce the effect of illumination's differences. 

Moreover the CNN converg faster on [0..1] data than on [0..255].
"""

# Normalize the data to make CNN faster
x_train = x_train / 255.0
x_test = x_test / 255.0

print(x_train.shape)
print(y_train.shape)

"""<a id="section-two-d"></a>
## 2.3 Reshape
"""

# Reshape image is 3D array (height = 28px, width = 28px , canal = 1)
x_train = x_train.reshape(-1,32,32,1)
x_test = x_test.reshape(-1,32,32,1)

print(x_train.shape)
print(y_train.shape)

print(x_test.shape)
print(y_test.shape)

"""Train and test images (32px x 32px) has been stock into pandas. We reshape all data to 32x32x1 3D matrices. 

Keras requires an extra dimension in the end which correspond to channels. 
"""

ra = np.random.randint(0, 13440, size=25)
plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.imshow(x_train[ra[i]].reshape(32,32).T,"gray")
    plt.xlabel(arabic_characters[int(y_train[ra[i]][0])])
plt.show()

"""<a id="section-two-e"></a>
## 2.5 Label encoding
"""

# Converting the class vector in integers to binary class matrix
y_train = to_categorical(y_train)
y_test = to_categorical(y_test)
print(y_train.shape, y_test.shape)
for i in zip(y_train[0], arabic_characters):
    print(i)

"""Labels are 28 characters of arabic_characters."""

print(y_train.shape)
print(y_test.shape)

"""# **PART A - DEEP LEARNING MODEL**

<a id="section-three"></a>
# 3. CNN

<a id="section-three-a"></a>
## 3.1 Define the model
"""

#Creating CNN model
def get_model(activationFn, lossFn, optimizer):
    In = Input(shape=(32,32,1))
    x = Conv2D(32, (5,5), padding="same", activation=activationFn)(In)
    x = Conv2D(32, (5,5), activation=activationFn)(x)
    x = Conv2D(32, (5,5), activation=activationFn)(x)
    x = MaxPooling2D((2,2))(x)
    x = BatchNormalization()(x)
    
    x = Conv2D(64, (5,5), padding="same", activation=activationFn)(x)
    x = Conv2D(64, (5,5), activation=activationFn)(x)
    x = Conv2D(64, (5,5), activation=activationFn)(x)
    x = MaxPooling2D((2,2))(x)
    x = BatchNormalization()(x)
    
    x = Flatten()(x)
    x = Dense(128, activation=activationFn)(x)
    x = Dense(128, activation=activationFn)(x)
    x = Dropout(0.4)(x)
    
    Out = Dense(28, activation="softmax")(x)
    
    model = Model(In, Out)
    model.compile(loss=lossFn, optimizer=optimizer, metrics=["accuracy"])
    return model

model = get_model("relu","categorical_crossentropy","adam")
keras.utils.vis_utils.plot_model(model, show_shapes=True, show_layer_names=True)

"""<a id="section-three-a"></a>
## 3.2 Data Augementation
"""

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 20

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

"""For the data augmentation, we chose to :
   - Randomly rotate some training images by 10 degrees
   - Randomly  Zoom by 10% some training images
   - Randomly shift images horizontally by 10% of the width
   - Randomly shift images vertically by 10% of the height
   
We did not apply a vertical_flip nor horizontal_flip since it could have lead to misclassify symetrical numbers such as 6 and 9.

Once our model is ready, we fit the training dataset .
"""

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

"""<a id="section-four"></a>
# 3.3. Evaluate the model

"""

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = model.predict(x_test)

# Evaluate model
score = model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""# **PART B - ACTIVATION FUNCTION**

We will now change the activation function **from reLU (Rectified Linear Unit) to seLU (Scaled Exponential Linear Unit)** and further observe, model efficiency.
"""

model = get_model("selu","categorical_crossentropy","adam")
keras.utils.vis_utils.plot_model(model, show_shapes=True, show_layer_names=True)

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 20

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = model.predict(x_test)

# Evaluate model
score = model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""After changing our activation function, the accuracy was decreased. Still, to further conclude the same, we will use another activation function.

We will now change our activation function from seLU to **Exponential linear unit (ELU)**.
"""

model = get_model("elu","categorical_crossentropy","adam")
keras.utils.vis_utils.plot_model(model, show_shapes=True, show_layer_names=True)

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 20

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = model.predict(x_test)

# Evaluate model
score = model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""After using reLU, seLU and ELU as our activation functions, its been observed that, **reLU** has the highest test accuracy as compared to others.

# **PART C - COST FUNCTION**

We will now change the activation function **from categorical_crossentropy to mean_squared_error** and further observe, model efficiency.
"""

model = get_model("relu","mean_squared_error","adam")
keras.utils.vis_utils.plot_model(model, show_shapes=True, show_layer_names=True)

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 20

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = model.predict(x_test)

# Evaluate model
score = model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""So, we can obseve that, after changing the cost function from cross-entropy to quadratic cost, our model's test accuracy was **decreased**.

Let's use another cost function - **hinge** to observe, how our model's accuracy is affected.
"""

model = get_model("relu","hinge","adam")
keras.utils.vis_utils.plot_model(model, show_shapes=True, show_layer_names=True)

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 20

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = model.predict(x_test)

# Evaluate model
score = model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""So, we can obseve that, after using cross-entropy, quadratic cost and hinge cost function, our model's test accuracy was best with **cross-entropy**.

# **PART D - EPOCHS**

Now, we have been using Epoch = 20, now we are going to first **double** the no. of epochs and then **reduce** the same by half, in order to observe, how our model's test accuracy changes.
"""

model = get_model("relu","categorical_crossentropy","adam")
keras.utils.vis_utils.plot_model(model, show_shapes=True, show_layer_names=True)

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 40

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = model.predict(x_test)

# Evaluate model
score = model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""We can observe that, our test accuracy was **increased** after doubling the no. of epochs from 20 to 40.

Now, lets see how the accuracy affects after changing the no of epochs to half, i.e 10.
"""

model = get_model("relu","categorical_crossentropy","adam")
keras.utils.vis_utils.plot_model(model, show_shapes=True, show_layer_names=True)

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 10

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = model.predict(x_test)

# Evaluate model
score = model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""So, we can see that, after reducing the no. of epochs, the test accuracy was also **decreased**.

# **PART E - GRADIENT ESTIMATION**

Now, we have been using 'adam' as our gradiant estimation. Now, lets change the gradiant estimation to '**rmsprop**' and further observe, model effciency.
"""

model = get_model("relu","categorical_crossentropy","rmsprop")
keras.utils.vis_utils.plot_model(model, show_shapes=True, show_layer_names=True)

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 40

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = model.predict(x_test)

# Evaluate model
score = model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""So, we can observe that, after using RMSprop as the gradiant estimator, test accuracy was **decreased**.

# **PART F - NETWORK ARCHITECTURE**

We will be reducing the no. of layers and further observe, how the test accuracy changes.
"""

#Creating CNN model
def get_network_model(activationFn, lossFn, optimizer):
    In = Input(shape=(32,32,1))
    x = Conv2D(32, (5,5), padding="same", activation=activationFn)(In)
    x = Conv2D(32, (5,5), activation=activationFn)(x)
    x = MaxPooling2D((2,2))(x)
    x = BatchNormalization()(x)
    
    x = Conv2D(64, (5,5), padding="same", activation=activationFn)(x)
    x = Conv2D(64, (5,5), activation=activationFn)(x)
    x = MaxPooling2D((2,2))(x)
    x = BatchNormalization()(x)
    
    x = Flatten()(x)
    x = Dense(128, activation=activationFn)(x)
    x = Dense(128, activation=activationFn)(x)
    x = Dropout(0.4)(x)
    
    Out = Dense(28, activation="softmax")(x)
    
    model = Model(In, Out)
    model.compile(loss=lossFn, optimizer=optimizer, metrics=["accuracy"])
    return model

network_model = get_network_model("relu","categorical_crossentropy","adam")
keras.utils.vis_utils.plot_model(network_model, show_shapes=True, show_layer_names=True)

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 40

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = network_model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = model.predict(x_test)

# Evaluate model
score = model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""So after reducing the no of layers, test accuracy was **decreased**.

# **PART G - NETWORK INITIALIZATION**

We will now change the network initialization to Glorot normal initializer, also called **Xavier** normal initializer and observe the test accuracy.
"""

#Creating CNN model
def get_network_initialization_model(activationFn, lossFn, optimizer):
    In = Input(shape=(32,32,1))
    x = Conv2D(32, (5,5), padding="same", activation=activationFn)(In)
    x = Conv2D(32, (5,5), activation=activationFn)(x)
    x = MaxPooling2D((2,2))(x)
    x = BatchNormalization()(x)
    
    x = Conv2D(64, (5,5), padding="same", activation=activationFn)(x)
    x = Conv2D(64, (5,5), activation=activationFn)(x)
    x = MaxPooling2D((2,2))(x)
    x = BatchNormalization()(x)
    
    x = Flatten()(x)
    x = Dense(128, activation=activationFn, kernel_initializer="glorot_normal")(x)
    x = Dense(128, activation=activationFn)(x)
    x = Dropout(0.4)(x)
    
    Out = Dense(28, activation="softmax")(x)
    
    model = Model(In, Out)
    model.compile(loss=lossFn, optimizer=optimizer, metrics=["accuracy"])
    return model

network_intialization_model = get_network_initialization_model("relu","categorical_crossentropy","adam")
keras.utils.vis_utils.plot_model(network_intialization_model, show_shapes=True, show_layer_names=True)

# With data augmentation to prevent overfitting
'''
Let's apply some data augmentation.

Data augmentation is a set of techniques used to generate new training samples from the original ones
by applying jitters and perturbations such that the classes labels are not changed.
In the context of computer vision, these random transformations can be translating,
rotating, scaling, shearing, flipping etc.

Data augmentation is a form of regularization because the training algorithm is being
constantly presented with new training samples,
allowing it to learn more robust and discriminative patterns
and reducing overfitting.
'''



datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

batch_size = 64
epochs = 40

train_gen = datagen.flow(x_train, y_train, batch_size=batch_size)
test_gen = datagen.flow(x_test, y_test, batch_size=batch_size)

model_checkpoint_callback = ModelCheckpoint(
    filepath="best.hdf5",
    monitor='val_accuracy', 
    verbose=1, 
    save_best_only=True, 
    mode='max')


history = network_intialization_model.fit_generator(train_gen, 
                              epochs = epochs,
                              verbose = 0,
                              steps_per_epoch = x_train.shape[0] // batch_size,
                              validation_data = test_gen,
                              validation_steps = x_test.shape[0] // batch_size,
                              callbacks=[model_checkpoint_callback])

# Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
plt.figure(figsize=(10,10))
plt.plot(history.history["accuracy"])
plt.plot(history.history["val_accuracy"])
plt.legend(["accuracy","val_accuracy"])
plt.show()

# predict results y_pred
y_pred = network_intialization_model.predict(x_test)

# Evaluate model
score = network_intialization_model.evaluate(x_test , y_test,verbose=3)

print('Test accuarcy: %2f%%' % round((score[1] * 100),5))

"""So, we can observe that, after changing the network initilizer to **Xavier**, our model's test accuracy was **increased**.

# **4. Conclusion**

So, after running our model through various scenarios, we can conclude that:

1. Activation functions, do affect the overall model's accuracy. As in our case, after using relu, selu and elu, we found that, **relu** had the best test accuracy.

2. Cost function also affects the test accuracy and after using Quadratic, cross-entropy and hinge, we found that, **cross-entropy** was best suited for our model. 

3. No. of epochs also affects the test accuracy and in our model, after **doubling** the no. of epochs, test accuracy was increased and reducing the no. of epochs, further decreased accuracy.

4. After changing the gradiant estimators, we found that with **adam**, our overall test accuracy was increased. Thus, we can say that, gradiant estimator also plays a significant role in model's performance.

5. We changed the no. of layers, and then concluded that, after decreasing the no. of layers, our model's test accuracy was also decreased.

6. After changing the network initilizer to **Xavier**, our model's test accuracy was increased. So, we can conclude that, network initializaion is also important for model's accuracy.

# **5. Refrence**

https://www.kaggle.com/mloey1/ahcd1/

https://www.kaggle.com/yehyachali/arabic-mnist-with-detection/notebook

https://keras.io/about/\n

Copyright 2021 <NAME>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
"""
# -*- coding: utf-8 -*- Deep_Learning_with_CNNs-1.ipynb  Automatically generated by Colaboratory.  Original file is located at     https://colab.research.google.com/drive/1_-DhPtcW-mhY1iI_T3vDDI2PMIKwx55H  <a id="section-one"></a>  # **1. Abstract**  The dataset is composed of 16800 Arabic characters written by 60 participants in the age range of 19-40.  We choose to build it with keras API (Tensorflow backend) which is very intuitive. Firstly, we will prepare the data (handwritten digits images) then we will focus on the CNN modeling and evaluation.  We achieved 99.671% of accuracy with CNN trained model.  **For computational reasons, we set the number of steps (epochs) to 2.**  This Notebook follows three main parts:  * The data preparation * The CNN modeling and evaluation * The results prediction and submission  # **2 Preparations** Preparing packages and data that will be used in the analysis process. Packages that will be loaded are mainly for data manipulation, data visualization and modeling. There are 2 datasets that are used in the analysis, they are train and test dataset. The main use of train dataset is to train models and use it to predict test dataset. While sample submission file is used to informed participants on the expected submission for the competition. # **2.1. Importing necessary Libraries** # Import the necessary libs # from keras.layers.normalization import BatchNormalization <a id="section-two"></a> # **2. Data preparation**  <a id="section-two-a"></a> ## 2.1 Load data # Commented out IPython magic to ensure Python compatibility. # %ls # loading the dataset Dataset has the following features:  Dataset size 13440 samples of handwritten images. The size of each image is 28x28 pixels.  Each image has only 1 color channel, i.e., grayscale image. Each pixel has value in the range of [0,255] where 0 represents black, and 255 represents white.  <a id="section-two-c"></a> ## 2.2 Normalization  We perform a grayscale normalization to reduce the effect of illumination's differences.   Moreover the CNN converg faster on [0..1] data than on [0..255]. # Normalize the data to make CNN faster <a id="section-two-d"></a> ## 2.3 Reshape # Reshape image is 3D array (height = 28px, width = 28px , canal = 1) Train and test images (32px x 32px) has been stock into pandas. We reshape all data to 32x32x1 3D matrices.   Keras requires an extra dimension in the end which correspond to channels. <a id="section-two-e"></a> ## 2.5 Label encoding # Converting the class vector in integers to binary class matrix Labels are 28 characters of arabic_characters. # **PART A - DEEP LEARNING MODEL**  <a id="section-three"></a> # 3. CNN  <a id="section-three-a"></a> ## 3.1 Define the model #Creating CNN model <a id="section-three-a"></a> ## 3.2 Data Augementation # With data augmentation to prevent overfitting Let's apply some data augmentation.  Data augmentation is a set of techniques used to generate new training samples from the original ones by applying jitters and perturbations such that the classes labels are not changed. In the context of computer vision, these random transformations can be translating, rotating, scaling, shearing, flipping etc.  Data augmentation is a form of regularization because the training algorithm is being constantly presented with new training samples, allowing it to learn more robust and discriminative patterns and reducing overfitting. # set input mean to 0 over the dataset # set each sample mean to 0 # divide inputs by std of the dataset # divide each input by its std # apply ZCA whitening # randomly rotate images in the range (degrees, 0 to 180) # Randomly zoom image # randomly shift images horizontally (fraction of total width) # randomly shift images vertically (fraction of total height) # randomly flip images # randomly flip images For the data augmentation, we chose to :    - Randomly rotate some training images by 10 degrees    - Randomly  Zoom by 10% some training images    - Randomly shift images horizontally by 10% of the width    - Randomly shift images vertically by 10% of the height     We did not apply a vertical_flip nor horizontal_flip since it could have lead to misclassify symetrical numbers such as 6 and 9.  Once our model is ready, we fit the training dataset . <a id="section-four"></a> # 3.3. Evaluate the model # Draw the loss and accuracy curves of the training set and the validation set. # Can judge whether it is under-fitting or over-fitting # predict results y_pred # Evaluate model # **PART B - ACTIVATION FUNCTION**  We will now change the activation function **from reLU (Rectified Linear Unit) to seLU (Scaled Exponential Linear Unit)** and further observe, model efficiency. # With data augmentation to prevent overfitting Let's apply some data augmentation.  Data augmentation is a set of techniques used to generate new training samples from the original ones by applying jitters and perturbations such that the classes labels are not changed. In the context of computer vision, these random transformations can be translating, rotating, scaling, shearing, flipping etc.  Data augmentation is a form of regularization because the training algorithm is being constantly presented with new training samples, allowing it to learn more robust and discriminative patterns and reducing overfitting. # set input mean to 0 over the dataset # set each sample mean to 0 # divide inputs by std of the dataset # divide each input by its std # apply ZCA whitening # randomly rotate images in the range (degrees, 0 to 180) # Randomly zoom image # randomly shift images horizontally (fraction of total width) # randomly shift images vertically (fraction of total height) # randomly flip images # randomly flip images # Draw the loss and accuracy curves of the training set and the validation set. # Can judge whether it is under-fitting or over-fitting # predict results y_pred # Evaluate model After changing our activation function, the accuracy was decreased. Still, to further conclude the same, we will use another activation function.  We will now change our activation function from seLU to **Exponential linear unit (ELU)**. # With data augmentation to prevent overfitting Let's apply some data augmentation.  Data augmentation is a set of techniques used to generate new training samples from the original ones by applying jitters and perturbations such that the classes labels are not changed. In the context of computer vision, these random transformations can be translating, rotating, scaling, shearing, flipping etc.  Data augmentation is a form of regularization because the training algorithm is being constantly presented with new training samples, allowing it to learn more robust and discriminative patterns and reducing overfitting. # set input mean to 0 over the dataset # set each sample mean to 0 # divide inputs by std of the dataset # divide each input by its std # apply ZCA whitening # randomly rotate images in the range (degrees, 0 to 180) # Randomly zoom image # randomly shift images horizontally (fraction of total width) # randomly shift images vertically (fraction of total height) # randomly flip images # randomly flip images # Draw the loss and accuracy curves of the training set and the validation set. # Can judge whether it is under-fitting or over-fitting # predict results y_pred # Evaluate model After using reLU, seLU and ELU as our activation functions, its been observed that, **reLU** has the highest test accuracy as compared to others.  # **PART C - COST FUNCTION**  We will now change the activation function **from categorical_crossentropy to mean_squared_error** and further observe, model efficiency. # With data augmentation to prevent overfitting Let's apply some data augmentation.  Data augmentation is a set of techniques used to generate new training samples from the original ones by applying jitters and perturbations such that the classes labels are not changed. In the context of computer vision, these random transformations can be translating, rotating, scaling, shearing, flipping etc.  Data augmentation is a form of regularization because the training algorithm is being constantly presented with new training samples, allowing it to learn more robust and discriminative patterns and reducing overfitting. # set input mean to 0 over the dataset # set each sample mean to 0 # divide inputs by std of the dataset # divide each input by its std # apply ZCA whitening # randomly rotate images in the range (degrees, 0 to 180) # Randomly zoom image # randomly shift images horizontally (fraction of total width) # randomly shift images vertically (fraction of total height) # randomly flip images # randomly flip images # Draw the loss and accuracy curves of the training set and the validation set. # Can judge whether it is under-fitting or over-fitting # predict results y_pred # Evaluate model So, we can obseve that, after changing the cost function from cross-entropy to quadratic cost, our model's test accuracy was **decreased**.  Let's use another cost function - **hinge** to observe, how our model's accuracy is affected. # With data augmentation to prevent overfitting Let's apply some data augmentation.  Data augmentation is a set of techniques used to generate new training samples from the original ones by applying jitters and perturbations such that the classes labels are not changed. In the context of computer vision, these random transformations can be translating, rotating, scaling, shearing, flipping etc.  Data augmentation is a form of regularization because the training algorithm is being constantly presented with new training samples, allowing it to learn more robust and discriminative patterns and reducing overfitting. # set input mean to 0 over the dataset # set each sample mean to 0 # divide inputs by std of the dataset # divide each input by its std # apply ZCA whitening # randomly rotate images in the range (degrees, 0 to 180) # Randomly zoom image # randomly shift images horizontally (fraction of total width) # randomly shift images vertically (fraction of total height) # randomly flip images # randomly flip images # Draw the loss and accuracy curves of the training set and the validation set. # Can judge whether it is under-fitting or over-fitting # predict results y_pred # Evaluate model So, we can obseve that, after using cross-entropy, quadratic cost and hinge cost function, our model's test accuracy was best with **cross-entropy**.  # **PART D - EPOCHS**  Now, we have been using Epoch = 20, now we are going to first **double** the no. of epochs and then **reduce** the same by half, in order to observe, how our model's test accuracy changes. # With data augmentation to prevent overfitting Let's apply some data augmentation.  Data augmentation is a set of techniques used to generate new training samples from the original ones by applying jitters and perturbations such that the classes labels are not changed. In the context of computer vision, these random transformations can be translating, rotating, scaling, shearing, flipping etc.  Data augmentation is a form of regularization because the training algorithm is being constantly presented with new training samples, allowing it to learn more robust and discriminative patterns and reducing overfitting. # set input mean to 0 over the dataset # set each sample mean to 0 # divide inputs by std of the dataset # divide each input by its std # apply ZCA whitening # randomly rotate images in the range (degrees, 0 to 180) # Randomly zoom image # randomly shift images horizontally (fraction of total width) # randomly shift images vertically (fraction of total height) # randomly flip images # randomly flip images # Draw the loss and accuracy curves of the training set and the validation set. # Can judge whether it is under-fitting or over-fitting # predict results y_pred # Evaluate model We can observe that, our test accuracy was **increased** after doubling the no. of epochs from 20 to 40.  Now, lets see how the accuracy affects after changing the no of epochs to half, i.e 10. # With data augmentation to prevent overfitting Let's apply some data augmentation.  Data augmentation is a set of techniques used to generate new training samples from the original ones by applying jitters and perturbations such that the classes labels are not changed. In the context of computer vision, these random transformations can be translating, rotating, scaling, shearing, flipping etc.  Data augmentation is a form of regularization because the training algorithm is being constantly presented with new training samples, allowing it to learn more robust and discriminative patterns and reducing overfitting. # set input mean to 0 over the dataset # set each sample mean to 0 # divide inputs by std of the dataset # divide each input by its std # apply ZCA whitening # randomly rotate images in the range (degrees, 0 to 180) # Randomly zoom image # randomly shift images horizontally (fraction of total width) # randomly shift images vertically (fraction of total height) # randomly flip images # randomly flip images # Draw the loss and accuracy curves of the training set and the validation set. # Can judge whether it is under-fitting or over-fitting # predict results y_pred # Evaluate model So, we can see that, after reducing the no. of epochs, the test accuracy was also **decreased**.  # **PART E - GRADIENT ESTIMATION**  Now, we have been using 'adam' as our gradiant estimation. Now, lets change the gradiant estimation to '**rmsprop**' and further observe, model effciency. # With data augmentation to prevent overfitting Let's apply some data augmentation.  Data augmentation is a set of techniques used to generate new training samples from the original ones by applying jitters and perturbations such that the classes labels are not changed. In the context of computer vision, these random transformations can be translating, rotating, scaling, shearing, flipping etc.  Data augmentation is a form of regularization because the training algorithm is being constantly presented with new training samples, allowing it to learn more robust and discriminative patterns and reducing overfitting. # set input mean to 0 over the dataset # set each sample mean to 0 # divide inputs by std of the dataset # divide each input by its std # apply ZCA whitening # randomly rotate images in the range (degrees, 0 to 180) # Randomly zoom image # randomly shift images horizontally (fraction of total width) # randomly shift images vertically (fraction of total height) # randomly flip images # randomly flip images # Draw the loss and accuracy curves of the training set and the validation set. # Can judge whether it is under-fitting or over-fitting # predict results y_pred # Evaluate model So, we can observe that, after using RMSprop as the gradiant estimator, test accuracy was **decreased**.  # **PART F - NETWORK ARCHITECTURE**  We will be reducing the no. of layers and further observe, how the test accuracy changes. #Creating CNN model # With data augmentation to prevent overfitting Let's apply some data augmentation.  Data augmentation is a set of techniques used to generate new training samples from the original ones by applying jitters and perturbations such that the classes labels are not changed. In the context of computer vision, these random transformations can be translating, rotating, scaling, shearing, flipping etc.  Data augmentation is a form of regularization because the training algorithm is being constantly presented with new training samples, allowing it to learn more robust and discriminative patterns and reducing overfitting. # set input mean to 0 over the dataset # set each sample mean to 0 # divide inputs by std of the dataset # divide each input by its std # apply ZCA whitening # randomly rotate images in the range (degrees, 0 to 180) # Randomly zoom image # randomly shift images horizontally (fraction of total width) # randomly shift images vertically (fraction of total height) # randomly flip images # randomly flip images # Draw the loss and accuracy curves of the training set and the validation set. # Can judge whether it is under-fitting or over-fitting # predict results y_pred # Evaluate model So after reducing the no of layers, test accuracy was **decreased**.  # **PART G - NETWORK INITIALIZATION**  We will now change the network initialization to Glorot normal initializer, also called **Xavier** normal initializer and observe the test accuracy. #Creating CNN model # With data augmentation to prevent overfitting Let's apply some data augmentation.  Data augmentation is a set of techniques used to generate new training samples from the original ones by applying jitters and perturbations such that the classes labels are not changed. In the context of computer vision, these random transformations can be translating, rotating, scaling, shearing, flipping etc.  Data augmentation is a form of regularization because the training algorithm is being constantly presented with new training samples, allowing it to learn more robust and discriminative patterns and reducing overfitting. # set input mean to 0 over the dataset # set each sample mean to 0 # divide inputs by std of the dataset # divide each input by its std # apply ZCA whitening # randomly rotate images in the range (degrees, 0 to 180) # Randomly zoom image # randomly shift images horizontally (fraction of total width) # randomly shift images vertically (fraction of total height) # randomly flip images # randomly flip images # Draw the loss and accuracy curves of the training set and the validation set. # Can judge whether it is under-fitting or over-fitting # predict results y_pred # Evaluate model So, we can observe that, after changing the network initilizer to **Xavier**, our model's test accuracy was **increased**.  # **4. Conclusion**  So, after running our model through various scenarios, we can conclude that:  1. Activation functions, do affect the overall model's accuracy. As in our case, after using relu, selu and elu, we found that, **relu** had the best test accuracy.  2. Cost function also affects the test accuracy and after using Quadratic, cross-entropy and hinge, we found that, **cross-entropy** was best suited for our model.   3. No. of epochs also affects the test accuracy and in our model, after **doubling** the no. of epochs, test accuracy was increased and reducing the no. of epochs, further decreased accuracy.  4. After changing the gradiant estimators, we found that with **adam**, our overall test accuracy was increased. Thus, we can say that, gradiant estimator also plays a significant role in model's performance.  5. We changed the no. of layers, and then concluded that, after decreasing the no. of layers, our model's test accuracy was also decreased.  6. After changing the network initilizer to **Xavier**, our model's test accuracy was increased. So, we can conclude that, network initializaion is also important for model's accuracy.  # **5. Refrence**  https://www.kaggle.com/mloey1/ahcd1/  https://www.kaggle.com/yehyachali/arabic-mnist-with-detection/notebook  https://keras.io/about/\n  Copyright 2021 <NAME>  Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:  The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.