<p>The result is not odd at all. The network has never learnt what makes your face special, but just remembered what makes the 500 set different from yours. Once you present a new face, it has no "memory" of it, and therefore interprets it as yours, just because none of features present in the 500 faces appears in the 501th.</p>

<p>Some ideas how to tackle this:</p>

<ul>
<li>Augment your data with <a href="https://keras.io/preprocessing/image/" rel="nofollow noreferrer">ImageDataGenerator</a>, as proposed by petezurich.</li>
<li>Increase your kernel size. 2*2 is too tiny to catch facial features. Consider 3*3, or even stacking two 3*3 in the first hidden layer.</li>
<li>Consider using Batch Normalization and regularization. Increase Dropout to 0.5.</li>
<li>Consider replacing pooling layers with <a href="https://arxiv.org/abs/1511.07122" rel="nofollow noreferrer">diluted convolutions</a> (available in Keras).</li>
<li>Make sure you normalise your input data.</li>
<li>Lower the number of feature maps (filters) in the first layer. Consider using e.g. 32 3*3 maps instead of 128 tiny elements (these are your main problem if I am to guess). This way you will force the network to generalise instead of learning some tiny nuances.</li>
</ul>

<p>A nice test for my hypothesis from the last point would be to visualise the activations in hidden layers, especially the first hidden layer. I have a feeling your network activates on some irrelevant features (or rather - noise), rather than "human features" (like eyes, haircut).</p>

<p>[EDIT after more code has been added]</p>

<ul>
<li>Centre your input data around zero.</li>
<li>Lower the batch size. With so few examples you don't want too much of averaging in a batch.</li>
</ul>

<p>I still think that using e.g. 16 or 32 filters in the first hidden layer should be the first thing to check. Look at your face. Can you spot 128 "features"? Unless you have some severe acne, I don't think so.</p>
CNN gives biased results
<p>I am using a binary classifier on CNN. I have two categories "me" and "others". I have around 250 images of myself and 500 of others ( random faces db ). My current implementation of layers is very simple</p>

<pre><code>    self.model.add(Conv2D(128, (2, 2), padding='same', 
    input_shape=dataset.X_train.shape[1:]))
    self.model.add(Activation('relu'))
    self.model.add(MaxPooling2D(pool_size=(2, 2)))
    self.model.add(Dropout(0.25))

    self.model.add(Conv2D(64, (2, 2), padding='same'))
    self.model.add(Activation('relu'))
    self.model.add(MaxPooling2D(pool_size=(2, 2)))
    self.model.add(Dropout(0.25))

    self.model.add(Conv2D(32, (1, 1), padding='same'))
    self.model.add(Activation('relu'))
    self.model.add(MaxPooling2D(pool_size=(2, 2)))
    self.model.add(Dropout(0.5))
    self.model.add(Dense(512))
    self.model.add(Activation('relu'))
    self.model.add(Dropout(0.25))
    self.model.add(Dense(2)) # for two classes
    self.model.add(Activation('softmax'))
</code></pre>

<p>My network reaches accuracy of 93% 
<a href="https://i.stack.imgur.com/8RVEV.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8RVEV.png" alt="enter image description here"></a></p>

<p><a href="https://i.stack.imgur.com/xxdmI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/xxdmI.png" alt="enter image description here"></a>
My problem is the when I use this network to predict faces it always recognises any face as mine. I have cropped the faces, applied gabor filter but nothing works. Any suggestion will be appreciated.</p>

<p>Prediction Results on Random Faces: [KK represents my face]
The probabilities are always over 97%:</p>

<pre><code>KK identified!
1/1 [==============================] - 0s
[[ 0.9741978  0.0258022]]
1/1 [==============================] - 0s

KK identified!
1/1 [==============================] - 0s
[[ 0.9897241   0.01027592]]
1/1 [==============================] - 0s
</code></pre>

<p>Prediction Results on my images: [KK represents my face]
The probabilities are always over 99%:</p>

<pre><code>KK identified!
1/1 [==============================] - 0s
[[ 0.99639165  0.00360837]]
1/1 [==============================] - 0s
KK identified!
1/1 [==============================] - 0s
[[ 0.99527925  0.00472075]]
1/1 [==============================] - 0s
</code></pre>

<p>Training code</p>

<pre><code>   def get_data(self, img_rows=IMAGE_SIZE, img_cols=IMAGE_SIZE, img_channels=3, nb_classes=2):

        images, labels = fetch_data('./data/')
        labels = np.reshape(labels, [-1])

        X_train, X_test, y_train, y_test =  \
            train_test_split(images, labels, test_size=0.3, random_state=random.randint(0, 100))
        X_valid, X_test, y_valid, y_test = \
            train_test_split(images, labels, test_size=0.3, random_state=random.randint(0, 100))
            #train_test_split(images, labels, test_size=0.3, random_state=np.random.seed(15))

        if K.image_dim_ordering() == 'th':
            X_train = X_train.reshape(X_train.shape[0], 3, img_rows, img_cols)
            X_valid = X_valid.reshape(X_valid.shape[0], 3, img_rows, img_cols)
            X_test = X_test.reshape(X_test.shape[0], 3, img_rows, img_cols)
            # input_shape = (3, img_rows, img_cols)
        else:
            X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 3)
            X_valid = X_valid.reshape(X_valid.shape[0], img_rows, img_cols, 3)
            X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 3)
            # input_shape = (img_rows, img_cols, 3)

        Y_train = np_utils.to_categorical(y_train, nb_classes)
        Y_valid = np_utils.to_categorical(y_valid, nb_classes)
        Y_test = np_utils.to_categorical(y_test, nb_classes)

        X_train = X_train.astype('float32')
        X_valid = X_valid.astype('float32')
        X_test = X_test.astype('float32')
        X_train /= 255
        X_valid /= 255
        X_test /= 255

        self.X_train = X_train
        self.X_valid = X_valid
        self.X_test = X_test
        self.Y_train = Y_train
        self.Y_valid = Y_valid
        self.Y_test = Y_test



def train_network(self, dataset, batch_size=32, nb_epoch=40, data_augmentation=True):
    sgd = SGD(lr=0.003, decay=0.0000001, momentum=0.9, nesterov=True)

    # adam = Adam(lr=0.01, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0001)
    self.model.compile(loss='binary_crossentropy',
                       optimizer=sgd,
                       metrics=['accuracy'])
    if not data_augmentation:
        processed_data = self.model.fit(dataset.X_train, dataset.Y_train,
                       batch_size=batch_size,
                       nb_epoch=nb_epoch,
                       validation_data=(dataset.X_valid, dataset.Y_valid),
                       shuffle=True)
    else:
        datagenerator = ImageDataGenerator(
            featurewise_center=False,
            samplewise_center=False,
            featurewise_std_normalization=False,
            samplewise_std_normalization=False,
            zca_whitening=False,
            rotation_range=20,
            width_shift_range=0.2,
            height_shift_range=0.2,
            horizontal_flip=True,
            vertical_flip=False)

        datagenerator.fit(dataset.X_train)

        processed_data = self.model.fit_generator(datagen.flow(dataset.X_train, dataset.Y_train, batch_size=batch_size, shuffle=True),
                                 samples_per_epoch=dataset.X_train.shape[0], nb_epoch=nb_epoch, validation_data=(dataset.X_valid, dataset.Y_valid))
</code></pre>

<p>Thanks</p>

<p>[Update: 11th June]</p>

<p>Layers</p>

<pre><code>def build_model(self, dataset, nb_classes=2):
        self.model = Sequential()

        self.model.add(Conv2D(32, (3, 3), padding='same', input_shape=dataset.X_train.shape[1:]))
        self.model.add(Activation('relu'))
        self.model.add(Conv2D(32, (3, 3)))
        self.model.add(Activation('relu'))
        self.model.add(MaxPooling2D(pool_size=(2, 2)))
        self.model.add(Dropout(0.5))

        self.model.add(Conv2D(16, (3, 3), padding='same'))
        self.model.add(Activation('relu'))
        self.model.add(Conv2D(16, (3, 3)))
        self.model.add(Activation('relu'))
        self.model.add(MaxPooling2D(pool_size=(2, 2)))
        self.model.add(Dropout(0.5))

        self.model.add(Flatten())
        self.model.add(Dense(512))
        self.model.add(Activation('relu'))
        self.model.add(Dropout(0.5))
        self.model.add(Dense(nb_classes))
        self.model.add(Activation('softmax'))

        self.model.summary()
</code></pre>

<p>Data augmentation</p>

<pre><code>    # this will do preprocessing and realtime data augmentation
    datagen = ImageDataGenerator(
        featurewise_center=True,             # set input mean to 0 over the dataset
        samplewise_center=False,              # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,   # divide each input by its std
        zca_whitening=False,                  # apply ZCA whitening
        rotation_range=20,                     # randomly rotate images in the range (degrees, 0 to 180)
        width_shift_range=0.2,                # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.2,               # randomly shift images vertically (fraction of total height)
        # rescale=1. / 255,
        # shear_range=0.2,
        # zoom_range=0.2,
        horizontal_flip=True,                 # randomly flip images
        vertical_flip=False)                  # randomly flip images

    datagen.fit(dataset.X_train)

    checkpoint = ModelCheckpoint(self.FILE_PATH, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
    callback_list = [checkpoint]

    # fit the model on the batches generated by datagen.flow()
    train_generator = datagen.flow(dataset.X_train, dataset.Y_train, batch_size=batch_size, shuffle=True)

    history = self.model.fit_generator(train_generator,
                                       samples_per_epoch=dataset.X_train.shape[0],
                                       nb_epoch=nb_epoch,
                                       validation_data=(dataset.X_valid, dataset.Y_valid),
                                       callbacks=callback_list)
</code></pre>

<p>Dataset</p>

<pre><code>class DataSet(object):

    def __init__(self):
        self.X_train = None
        self.X_valid = None
        self.X_test = None
        self.Y_train = None
        self.Y_valid = None
        self.Y_test = None

    # support only binary classification for now, thus 2 class limit
    def get_data(self, img_rows=IMAGE_SIZE, img_cols=IMAGE_SIZE, img_channels=3, nb_classes=2):

        images, labels = fetch_data('./data/')
        labels = np.reshape(labels, [-1])

        X_train, X_test, y_train, y_test =  \
            train_test_split(images, labels, test_size=0.2, random_state=random.randint(0, 100))

        X_valid, X_test, y_valid, y_test = \
            train_test_split(images, labels, test_size=0.2, random_state=random.randint(0, 100))

        if K.image_dim_ordering() == 'th':
            X_train = X_train.reshape(X_train.shape[0], 3, img_rows, img_cols)
            X_valid = X_valid.reshape(X_valid.shape[0], 3, img_rows, img_cols)
            X_test = X_test.reshape(X_test.shape[0], 3, img_rows, img_cols)

        else:
            X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 3)
            X_valid = X_valid.reshape(X_valid.shape[0], img_rows, img_cols, 3)
            X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 3)


        # convert class vectors to binary class matrices
        Y_train = np_utils.to_categorical(y_train, nb_classes)
        Y_valid = np_utils.to_categorical(y_valid, nb_classes)
        Y_test = np_utils.to_categorical(y_test, nb_classes)

        X_train = X_train.astype('float32')
        X_valid = X_valid.astype('float32')
        X_test = X_test.astype('float32')
        X_train /= 255
        X_valid /= 255
        X_test /= 255

        self.X_train = X_train
        self.X_valid = X_valid
        self.X_test = X_test
        self.Y_train = Y_train
        self.Y_valid = Y_valid
        self.Y_test = Y_test
</code></pre>