<p>There are many ways to prevent overfitting, according to the <strong>papers</strong> below:</p>
<ul>
<li>Dropout layers (Disabling randomly neurons). <a href="https://www.cs.toronto.edu/%7Ehinton/absps/JMLRdropout.pdf" rel="nofollow noreferrer">https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf</a></li>
<li>Input Noise (e.g. Random Gaussian Noise on the imges). <a href="https://arxiv.org/pdf/2010.07532.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2010.07532.pdf</a></li>
<li>Random Data Augmentations (e.g. Rotating, Shifting, Scaling, etc.).
<a href="https://arxiv.org/pdf/1906.11052.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1906.11052.pdf</a></li>
<li>Adjusting Number of Layers &amp; Units.
<a href="https://clgiles.ist.psu.edu/papers/UMD-CS-TR-3617.what.size.neural.net.to.use.pdf" rel="nofollow noreferrer">https://clgiles.ist.psu.edu/papers/UMD-CS-TR-3617.what.size.neural.net.to.use.pdf</a></li>
<li>Regularization Functions (e.g. L1, L2, etc)
<a href="https://www.researchgate.net/publication/329150256_A_Comparison_of_Regularization_Techniques_in_Deep_Neural_Networks" rel="nofollow noreferrer">https://www.researchgate.net/publication/329150256_A_Comparison_of_Regularization_Techniques_in_Deep_Neural_Networks</a></li>
<li>Early Stopping: <strong>If you notice that for N successive epochs that your model's training loss is decreasing, but the model performs poorly on validaiton data set, then It is a good sign to stop the training.</strong></li>
<li>Shuffling the training data or K-Fold cross validation is also common way way of dealing with Overfitting.</li>
</ul>
<p>I found this great repository, which contains examples of how to implement data augmentations:
<a href="https://github.com/kochlisGit/random-data-augmentations" rel="nofollow noreferrer">https://github.com/kochlisGit/random-data-augmentations</a></p>
<p>Also, this repository here seems to have examples of CNNs that implement most of the above methods:
<a href="https://github.com/kochlisGit/Tensorflow-State-of-the-Art-Neural-Networks" rel="nofollow noreferrer">https://github.com/kochlisGit/Tensorflow-State-of-the-Art-Neural-Networks</a></p>
CNN + LSTM model for images performs poorly on validation data set
<p>My training and loss curves look like below and yes, similar graphs have received comments like &quot;Classic overfitting&quot; and I get it.</p>
<p><a href="https://i.stack.imgur.com/6u8ey.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/6u8ey.png" alt="training and Lossurves" /></a></p>
<p>My model looks like below,</p>
<pre><code>input_shape_0 = keras.Input(shape=(3,100, 100, 1), name=&quot;img3&quot;)

model = tf.keras.layers.TimeDistributed(Conv2D(8, 3, activation=&quot;relu&quot;))(input_shape_0)
model = tf.keras.layers.TimeDistributed(Dropout(0.3))(model)
model = tf.keras.layers.TimeDistributed(MaxPooling2D(2))(model)

model = tf.keras.layers.TimeDistributed(Conv2D(16, 3, activation=&quot;relu&quot;))(model)
model = tf.keras.layers.TimeDistributed(MaxPooling2D(2))(model)

model = tf.keras.layers.TimeDistributed(Conv2D(32, 3, activation=&quot;relu&quot;))(model)
model = tf.keras.layers.TimeDistributed(MaxPooling2D(2))(model)

model = tf.keras.layers.TimeDistributed(Dropout(0.3))(model)
model = tf.keras.layers.TimeDistributed(Flatten())(model)
model = tf.keras.layers.TimeDistributed(Dropout(0.4))(model)

model = LSTM(16, kernel_regularizer=tf.keras.regularizers.l2(0.007))(model)

# model = Dense(100, activation=&quot;relu&quot;)(model)
# model = Dense(200, activation=&quot;relu&quot;,kernel_regularizer=tf.keras.regularizers.l2(0.001))(model)
model = Dense(60, activation=&quot;relu&quot;)(model)
# model = Flatten()(model)

model = Dropout(0.15)(model)
out = Dense(30, activation='softmax')(model)

model = keras.Model(inputs=input_shape_0, outputs = out, name=&quot;mergedModel&quot;)

def get_lr_metric(optimizer):
    def lr(y_true, y_pred):
        return optimizer.lr
    return lr

opt = tf.keras.optimizers.RMSprop()
lr_metric = get_lr_metric(opt)
# merged.compile(loss='sparse_categorical_crossentropy', 
                 optimizer='adam', metrics=['accuracy'])
model.compile(loss='sparse_categorical_crossentropy', 
                optimizer=opt, metrics=['accuracy',lr_metric])
model.summary()
</code></pre>
<p>In the above model building code, please consider the commented lines as some of the approaches I have tried so far.</p>
<p>I have followed the suggestions given as answers and comments to this kind of question and none seems to be working for me. Maybe I am missing something really important?</p>
<p>Things that I have tried:</p>
<ol>
<li>Dropouts at different places and different amounts.</li>
<li>Played with inclusion and expulsion of dense layers and their number of units.</li>
<li>Number of units on the LSTM layer was tried with different values (started from as low as 1 and now at 16, I have the best performance.)</li>
<li>Came across weight regularization techniques and tried to implement them as shown in the code above and so tried to put it at different layers ( I need to know what is the technique in which I need to use it instead of simple trial and error - this is what I did and it seems wrong)</li>
<li>Implemented learning rate scheduler using which I reduce the learning rate as the epochs progress after a certain number of epochs.</li>
<li>Tried two LSTM layers with the first one having return_sequences = true.</li>
</ol>
<p>After all these, I still cannot overcome the overfitting problem.
My data set is properly shuffled and divided in a train/val ratio of 80/20.</p>
<p>Data augmentation is one more thing that I found commonly suggested which I am yet to try, but I want to see if I am making some mistake so far which I can correct it and avoid diving into data augmentation steps for now. My data set has the below sizes:</p>
<pre><code>Training images: 6780
Validation images: 1484
</code></pre>
<p>The numbers shown are samples and each sample will have 3 images.  So basically, I input 3 mages at once as one sample to my time-distributed <code>CNN</code> which is then followed by other layers as shown in the model description. Following that, my training images are 6780 * 3 and my Validation images are 1484 * 3. Each image is 100 * 100 and is on channel 1.</p>
<p>I am using <code>RMS prop</code> as the optimizer which performed better than <code>adam</code> as per my testing</p>
<p><strong>UPDATE</strong></p>
<p>I tried some different architectures and some reularizations and dropouts at different places and I am now able to achieve a val_acc of 59% below is the new model.</p>
<pre><code>#  kernel_regularizer=tf.keras.regularizers.l2(0.004)
# kernel_constraint=max_norm(3)
model = tf.keras.layers.TimeDistributed(Conv2D(32, 3, activation=&quot;relu&quot;))(input_shape_0)
model = tf.keras.layers.TimeDistributed(Dropout(0.3))(model)
model = tf.keras.layers.TimeDistributed(MaxPooling2D(2))(model)

model = tf.keras.layers.TimeDistributed(Conv2D(64, 3, activation=&quot;relu&quot;))(model)
model = tf.keras.layers.TimeDistributed(MaxPooling2D(2))(model)

model = tf.keras.layers.TimeDistributed(Conv2D(128, 3, activation=&quot;relu&quot;))(model)
model = tf.keras.layers.TimeDistributed(MaxPooling2D(2))(model)


model = tf.keras.layers.TimeDistributed(Dropout(0.3))(model)

model = tf.keras.layers.TimeDistributed(GlobalAveragePooling2D())(model)

model = LSTM(128, return_sequences=True,kernel_regularizer=tf.keras.regularizers.l2(0.040))(model)
model = Dropout(0.60)(model)
model = LSTM(128, return_sequences=False)(model)
model = Dropout(0.50)(model)
out = Dense(30, activation='softmax')(model)
</code></pre>