<p>There are at least 5 reasons which might cause such behavior:</p>

<ol>
<li><p><strong>Outliers:</strong> imagine that you have 10 exactly the same images and 9 out of them belong to class <strong>A</strong> and one belongs to class <strong>B</strong>. In this case, a model will start to assign a high probability of class <strong>A</strong> to this example because of the majority of examples. But then - a signal from outlier might destabilize model and make accuracy decreasing.  In theory, a model should stabilize at assigning score <strong>90%</strong> to class <strong>A</strong> but it might last many epochs.</p>

<p><strong>Solutions:</strong> In order to deal with such examples I advise you to use <em>gradient clipping</em> (you may add such option in your optimizer). If you want to check if this phenomenon occurs - you may check your losses distributions (losses of individual examples from training set) and look for outliers.</p></li>
<li><p><strong>Bias</strong>: Now imagine that you have 10 exactly the same images but 5 of them have assigned class <strong>A</strong> and 5 - class <strong>B</strong>. In this case, a model will try to assign approximately <strong>50%-50%</strong> distribution on both of these classes. Now - your model can achieve at most 50% of accuracy here - choosing one class out of two valid.</p>

<p><strong>Solution:</strong> Try to increase the model capacity - very often you have a set of really similar images - adding expressive power might help to discriminate similar examples. Beware of overfitting though. Another solution is to try <a href="https://keras.io/callbacks/#reducelronplateau" rel="noreferrer">this</a> strategy in your training. If you want to check if such phenomenon occurs - check the distribution of losses of individual examples. If a distribution would be skewed toward higher values - you are probably suffering from <strong>bias</strong>.</p></li>
<li><p><strong>Class inbalance</strong>: Now imagine that <strong>90%</strong> of your images belong to class <strong>A</strong>. In an early stage of your training, your model is mainly concentrating on assigning this class to almost all of examples. This might make individual losses to achieve really high values and destabilize your model by making a predicted distribution more unstable. </p>

<p><strong>Solution:</strong> once again - gradient clipping. Second thing - patience, try simply leaving your model for more epochs. A model should learn more subtle in a further phase of training. And of course - try class balancing - by either assigning <code>sample_weights</code> or <code>class_weights</code>. If you want to check if this phenomenon occurs - check your class distribution.</p></li>
<li><p><strong>Too strong regularization:</strong> if you set your regularization to be too strict - a training process is mainly concentrated on making your weights to have smaller norm than actually learning interesting insights.</p>

<p><strong>Solution:</strong> add a <code>categorical_crossentropy</code> as a metric and observe if it's also decreasing. If not - then it means that your regularization is too strict - try to assign less weight penalty then.</p></li>
<li><p><strong>Bad model design</strong> - such behavior might be caused by a wrong model design. There are several good practices which one might apply in order to improve your model:</p>

<p><strong>Batch Normalization</strong> - thanks to this technique you are preventing your model from radical changes of inner network activations. This makes training much more stable and efficient. With a small batch size, this might be also a genuine way of regularizing your model.</p>

<p><strong>Gradient clipping</strong> - this makes your model training much more stable and efficient.</p>

<p><strong>Reduce bottleneck effect</strong> - read <a href="https://arxiv.org/abs/1512.00567" rel="noreferrer">this</a> fantastic paper and check if your model might suffer from <strong>bottleneck problem</strong>.</p>

<p><strong>Add auxiliary classifiers</strong> - if you are training your network from scratch - this should make your features much more meaningful and your training - faster and more efficient.</p></li>
</ol>
Relationship between loss and accuracy
<p>Is it practically possible to have decreasing loss and decreasing accuracy at each epoch when training a CNN model?
I am getting the below result while training.</p>

<p><a href="https://i.stack.imgur.com/LL9k4.png" rel="noreferrer"><img src="https://i.stack.imgur.com/LL9k4.png" alt="enter image description here"></a></p>

<p>Can someone explain the possible reasons why this is happening?</p>