<p>I faced a similar problem and managed to fix it by removing the Batch Normalisation layer that's just before the output dense layer. This made a ton of difference. Also one of the suggestions I was given is to remove the Dropout layer as it might be causing Shift Variance. Check this <a href="https://arxiv.org/pdf/1801.05134.pdf" rel="nofollow noreferrer">paper</a></p>
<p>I got part of the solution from this <a href="https://stackoverflow.com/questions/39691902/ordering-of-batch-normalization-and-dropout?noredirect=1&amp;lq=1">thread</a>.</p>
Keras: Training loss decrases (accuracy increase) while validation loss increases (accuracy decrease)
<p>I am working on a very sparse dataset with the point of predicting 6 classes. 
I have tried working with a lot of models and architectures, but the problem remains the same. </p>

<p>When I start training, the acc for training will slowly start to increase and loss will decrease where as the validation will do the exact opposite. </p>

<p>I have <strong>really tried</strong> to deal with overfitting, and I simply cannot still believe that this is what is coursing this issue. </p>

<h2>What have I tried</h2>

<p>Transfer learning on VGG16:</p>

<ul>
<li>exclude top layer and add dense layer with 256 units and 6 units softmax output layer</li>
<li>finetune the top CNN block</li>
<li>finetune the top 3-4 CNN blocks</li>
</ul>

<p>To deal with overfitting I use heavy augmentation in Keras and dropout after the 256 dense layer with p=0.5.</p>

<p>Creating own CNN with VGG16-ish architecture:</p>

<ul>
<li>including batch normalization wherever possible</li>
<li>L2 regularization on each CNN+dense layer</li>
<li>Dropout from anywhere between 0.5-0.8 after each CNN+dense+pooling layer</li>
<li>Heavy data augmentation in "on the fly" in Keras</li>
</ul>

<p>Realising that perhaps I have too many free parameters:</p>

<ul>
<li>decreasing the network to only contain 2 CNN blocks + dense + output. </li>
<li>dealing with overfitting in the same manner as above.</li>
</ul>

<p>Without exception <strong>all</strong> training sessions are looking like this:
<a href="https://i.stack.imgur.com/Vnwhi.png" rel="noreferrer">Training &amp; Validation loss+accuracy</a></p>

<p>The last mentioned architecture looks like this:</p>

<pre><code>    reg = 0.0001

    model = Sequential()

    model.add(Conv2D(8, (3, 3), input_shape=input_shape, padding='same',
            kernel_regularizer=regularizers.l2(reg)))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Dropout(0.7))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.5))

    model.add(Conv2D(16, (3, 3), input_shape=input_shape, padding='same',
            kernel_regularizer=regularizers.l2(reg)))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Dropout(0.7))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.5))

    model.add(Flatten())
    model.add(Dense(16, kernel_regularizer=regularizers.l2(reg)))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Dropout(0.5))

    model.add(Dense(6))
    model.add(Activation('softmax'))

    model.compile(loss='categorical_crossentropy', optimizer='SGD',metrics=['accuracy'])
</code></pre>

<p>And the data is augmented by the generator in Keras and is loaded with flow_from_directory:</p>

<pre><code>    train_datagen = ImageDataGenerator(rotation_range=10,
                                width_shift_range=0.05,
                                height_shift_range=0.05,
                                shear_range=0.05,
                                zoom_range=0.05,
                                rescale=1/255.,
                                fill_mode='nearest',
                                channel_shift_range=0.2*255)
    train_generator = train_datagen.flow_from_directory(
                train_data_dir,
                target_size=(img_width, img_height),
                batch_size=batch_size,
                shuffle = True,
                class_mode='categorical')

    validation_datagen = ImageDataGenerator(rescale=1/255.)
    validation_generator = validation_datagen.flow_from_directory(
                                            validation_data_dir,
                                            target_size=(img_width, img_height),
                                            batch_size=1,
                                            shuffle = True,
                                            class_mode='categorical')
</code></pre>