<p>There are different things going on here.</p>

<h2>Class imbalance</h2>

<p>As already discussed in the comments, your training set consists of 2941  negative samples (faces from other persons than you) and 308 positive samples (images with your face on it). So as classifier which always votes for the negative class gets 2941 of the 3249 samples right, i.e. 90.5%.
Your training accuracy score should be read with this information in mind, because your 89.4% hint that your network has learnt nothing valuable.</p>

<p>What you could try: </p>

<ol>
<li>Oversample the minority class</li>
<li>Undersample the majority class</li>
<li>Use weights in your loss function</li>
</ol>

<p>There exists a vast body of literature and tutorials on this topic, so just look around a bit.</p>

<h2>Model capacity/network design</h2>

<p>You mentioned in the comments that after using balanced training data, the loss still doesn't decrease. This hints that the capacity of your model is not high enough or that your model is not well designed for that task.</p>

<p>There exist many different network structures for image processing tasks, depending on what exactly the task is. For example, a network for doing pixel-wise classification and instance segmentation (like <a href="https://arxiv.org/abs/1703.06870" rel="nofollow noreferrer">Mask RCNN</a>) will look different from a network that is designed to do face recognition (like <a href="https://arxiv.org/abs/1503.03832" rel="nofollow noreferrer">FaceNet</a>). Even if we only focus on one task (in this/your case: face recognition) the network structures you find will look very different.</p>

<p>So what does that mean? Building a neural network for a specific task is not trivial and it is not easy to predict which structures work well for which task. A lot of trial and error is often needed, to find a suitable structure. </p>

<p>The model capacity describes how "powerful" your model is. There is no rigorous definition for this but you can think about it like this: Different tasks can be harder or easier to solve. A certain model can only solve problems which are "easy enough" and on more difficult problems the model will only show poor performance. The model capacity often goes hand in hand with the network design/structure.</p>

<p>How does that help you? Well, look at different network structures that do the same task as you aim to do and understand why certain building blocks are used and what they are good for. Then you can try to rebuild this structures and experiment with it. A good starting point might be <a href="https://hackernoon.com/building-a-facial-recognition-pipeline-with-deep-learning-in-tensorflow-66e7645015b8" rel="nofollow noreferrer">this</a> or <a href="https://medium.com/@ageitgey/machine-learning-is-fun-part-4-modern-face-recognition-with-deep-learning-c3cffc121d78" rel="nofollow noreferrer">this</a> tutorial on face recognition.</p>
Face classification CNN in Keras does not train properly
<p>i am fairly new to machine learning and this is my first post on StackOverflow.
I want to train a CNN such that it can distinguish my face from others.
My model just stops improving after the first 3 epochs.</p>

<p>I found some faces in online databases for machine learning, then centered, grayscaled and cropped them. The same thing i did with pictures taken by my webcam from my face.</p>

<p>The data for the NN looks like this:</p>

<pre>
| input                 | output    |       |
|---------------------- |--------   |---    |
| face of me            | 0         | 1     |
| face of someone else  | 1         | 0     |
</pre>

<p>So far so good. </p>

<p>I then tried to train the CNN with this with the following structure:</p>

<pre><code>model= Sequential()

# sort out the input layer later
model.add(Conv2D(32,3,3, activation='relu', input_shape=(100,100,3)))
model.add(MaxPooling2D((2,2)))

model.add(Conv2D(12,3,3, activation='relu'))
model.add(MaxPooling2D((2,2)))

model.add(Flatten())
model.add(Dense(600, activation='relu'))
model.add(Dense(60, activation='relu'))
#model.add(Dropout(p=0.2))
model.add(Dense(2, activation='softmax'))
model.summary()
epochs = 100
lrate = 0.01
decay = lrate/epochs
sgd = SGD(lr=lrate, momentum=0.9, decay=decay, nesterov=False)
#%%
model.compile(loss='categorical_crossentropy', optimizer="adam", metrics=['accuracy'])
</code></pre>

<p>I tried different optimizers (sdg,adam,rmsprop) and had some attempts on other parameters for the CNN but every time the model just stops reducing the loss after the first few epochs.</p>

<pre><code>Epoch 1/100
2176/2176 [==============================] - 55s 25ms/step - loss: 1.8043 - acc: 0.8869
Epoch 2/100
2176/2176 [==============================] - 56s 26ms/step - loss: 1.7037 - acc: 0.8943
Epoch 3/100
2176/2176 [==============================] - 57s 26ms/step - loss: 1.7037 - acc: 0.8943
Epoch 4/100
</code></pre>

<p>No improvement in Loss after epoch 2 for this example.</p>

<p>Do you have any idea why this might be the case?</p>