<p>To start I would recommend using stride 1, not two in your first layer. In the first two layers (conv2d and maxpool) you are already downsampling the image to 16x16, and the network hasn't had a chance to do much. You want to have at least a few layers for each unique number of filters before you downsample.</p>
<p>E.g architecture that might work better:</p>
<p>(conv2d 64 filters, stride 1) x 3</p>
<p>conv2d 128 filters, stride 2</p>
<p>(conv2d 128 filters, stride 1) x 3</p>
<p>conv2d 256 filters, stride 2</p>
<p>(conv2d 256 filters, stride 1) x 3</p>
<p>flatten</p>
<p>dense layers</p>
<p>For more ideas for architecture design, I recommend looking at models such as VGG:
<a href="https://arxiv.org/pdf/1409.1556.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1409.1556.pdf</a> (page 3).</p>
<p>You won't be able to verbatim copy those since your data is smaller, but note how they have more convolutional layers and less dense layers. They also don't downsample as harshly in the beginning.</p>
<p>I'm also curious as to the size of your dataset, and what is your training/validation/testing split. Do you successfully get close to 100% accuracy on the training data?</p>
Increase Accuracy on 2D CNN
<p>I've been trying to train a 2D CNN for an image classification problem. My data consists of 64 by 64 pixel images each labeled with a number from 1-37. I have my CNN architecture below:</p>
<pre><code>train_dataset = train.flow_from_directory('/kaggle/input/temp-frames/frames/train', target_size=(64,64), batch_size=256, class_mode='categorical')
validation_dataset = train.flow_from_directory('/kaggle/input/temp-frames/frames/validation', target_size=(64,64), batch_size=256, class_mode='categorical')

model = Sequential()
model.add(Conv2D(filters= 64, kernel_size=(3,3), activation ='relu',strides = (2,2), padding = 'valid', input_shape= (64,64,3)))
model.add(MaxPooling2D(pool_size=(2,2), padding='same'))

model.add(Flatten())

model.add(Dense(1024, activation='relu'))
model.add(Dropout(.5))
model.add(Dense(1024, activation='relu'))
model.add(Dropout(.5))
model.add(Dense(1024, activation='relu'))
model.add(Dropout(.5))

model.add(Dense(37)) 
model.add(Activation('softmax'))

optimizer = keras.optimizers.Adam(lr=0.01)

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])


history = model.fit(train_dataset, epochs = 100, batch_size = 32, validation_data = validation_dataset, shuffle = True)
</code></pre>
<p>For some reason, my 2D CNN(get accuracy of 16%) performs worse than my 1D CNN(gets accuracy of 30%). I am wondering if there is any way to improve my model to achieve better results.</p>