<p>Fashion MNIST is a harder problem than MNIST. Therefore, it is not surprising that your architecture does not perform as good.</p>

<p>If you want to achieve a higher accuracy, you may want to try a method described in <a href="https://arxiv.org/pdf/1708.04896.pdf" rel="nofollow noreferrer">this paper</a>.</p>
Why i can't improve the performance of a CNN on the Fashion MNIST dataset?
<p>I am playing around with Pytorch and i implemented a CNN on MNIST dataset which has 99+% accuracy on both train and test sets.</p>

<p>I decided to switch to Fashion MNIST in order to see how the architecture of my network performs.  I got 95% accuracy on the train set and 91% on test set.</p>

<p>Then, i started trying to improve that performance by tuning the model.</p>

<p>Briefly, my model looks like this: </p>

<pre><code>         Conv -&gt; ReLU -&gt; Batch norm -&gt; Max pool -&gt;
         Conv -&gt; ReLU -&gt; Batch norm -&gt; Max pool -&gt;
         Conv -&gt; ReLU -&gt; Batch norm -&gt; Max pool -&gt;
         Conv -&gt; ReLU -&gt; Batch norm -&gt; Max pool -&gt;
         Linear -&gt; ReLu -&gt; Linear -&gt; Output

         Optimizer: Stochastic Gradient Descent
         Transformations: ToTensor() only
</code></pre>

<p>My tests where removing the last Conv layer, adding average pooling instead of max pooling in the last Conv layer, inspecting the train loss curve in order to adjust the learning rate statically or dynamically and change the batch size.</p>

<p>However with the above combinations either my model will overfit (eg. 97% train, 89% test) or it will not have the best performance (eg 91% train, 89% test).</p>

<p>Am i missing something?  Am i doing something wrong?  Are there any other tuning parameters that i need to adjust that i didn't think of?</p>

<p>Thank you</p>