<p><code>fastai</code> is using a lot of tricks under the hood. A quick catch of what they're doing and you're not.</p>
<p><strong>Those are in the order that I think matters most</strong>, especially the first two should improve your scores.</p>
<h1>TLDR</h1>
<p>Use some scheduler (<a href="https://pytorch.org/docs/1.6.0/optim.html#torch.optim.lr_scheduler.CyclicLR" rel="noreferrer">torch.optim.lr_scheduler.CyclicLR</a> preferably) and <code>AdamW</code> instead of <code>SGD</code>.</p>
<h1>Longer version</h1>
<h2><a href="https://fastai1.fast.ai/basic_train.html#fit_one_cycle" rel="noreferrer"><code>fit_one_cycle</code></a></h2>
<p><a href="https://fastai1.fast.ai/callbacks.one_cycle.html" rel="noreferrer">1 cycle policy by Leslie Smith</a> is used in <code>fastai</code>. In PyTorch one can create similar routine using <a href="https://pytorch.org/docs/1.6.0/optim.html#torch.optim.lr_scheduler.CyclicLR" rel="noreferrer">torch.optim.lr_scheduler.CyclicLR</a> but that would require some manual setup.</p>
<p>Basically it starts with lower learning rate, gradually increases up to <code>5e-3</code> in your case and comes back to lower learning rate again (making a cycle). You can adjust how the <code>lr</code> should raise and fall (in <code>fastai</code> it does so using cosine annealing IIRC).</p>
<p>Your learning rate is too high at the beginning, some scheduler should help, test it out first of all.</p>
<h2>Optimizer</h2>
<p>In the provided code snippet you use <code>torch.optim.SGD</code> (as <code>optim_fn</code> is <code>None</code> and default is set) which is harder to setup correctly (usually).</p>
<p>On the other hand, if you manage to set it up manually correctly, you might generalize better.</p>
<p>Also <code>fastai</code> <strong>does not use <code>Adam</code> by default!</strong> It uses <a href="https://pytorch.org/docs/1.6.0/optim.html#torch.optim.AdamW" rel="noreferrer"><code>AdamW</code></a> if <code>true_wd</code> is set (I think, it will be default in your case anyway, see <a href="https://github.com/fastai/fastai1/blob/master/fastai/basic_train.py#L148" rel="noreferrer">source code</a>). <code>AdamW</code> decouples weight decay from adaptive learning rate which should improve convergence (read <a href="https://towardsdatascience.com/why-adamw-matters-736223f31b5d" rel="noreferrer">here</a> or <a href="https://arxiv.org/abs/1711.05101" rel="noreferrer">original paper</a></p>
<h2>Number of epochs</h2>
<p>Set the same number of epochs if you want to compare both approaches, currently it's apple to oranges.</p>
<h2>Gradient clipping</h2>
<p>You do not clip gradient (it is commented out), might help or not depending on the task. Would not focus on that one for now tbh.</p>
<h2>Other tricks</h2>
<p>Read about <a href="https://fastai1.fast.ai/basic_train.html#Learner" rel="noreferrer">Learner</a> and <a href="https://fastai1.fast.ai/basic_train.html#fit_one_cycle" rel="noreferrer">fit_one_cycle</a> and try to setup something similar in PyTorch (rough guidelines described above)</p>
<p>Also you might use some form of data augmentation to improve the scores even further, but that's out of the question's scope I suppose.</p>
Training results are different for Classification using Pytorch APIs and Fast-ai
<p>I have two training python scripts. One using Pytorch's API for classification training and another one is using Fast-ai.
Using Fast-ai has much better results.</p>
<p>Training outcomes are as follows.</p>
<pre><code>Fastai
epoch     train_loss  valid_loss  accuracy  time    
0         0.205338    2.318084    0.466482  23:02                         
1         0.182328    0.041315    0.993334  22:51                         
2         0.112462    0.064061    0.988932  22:47                         
3         0.052034    0.044727    0.986920  22:45                         
4         0.178388    0.081247    0.980883  22:45                         
5         0.009298    0.011817    0.996730  22:44                         
6         0.004008    0.003211    0.999748  22:43 

Using Pytorch
Epoch [1/10], train_loss : 31.0000 , val_loss : 1.6594, accuracy: 0.3568
Epoch [2/10], train_loss : 7.0000 , val_loss : 1.7065, accuracy: 0.3723
Epoch [3/10], train_loss : 4.0000 , val_loss : 1.6878, accuracy: 0.3889
Epoch [4/10], train_loss : 3.0000 , val_loss : 1.7054, accuracy: 0.4066
Epoch [5/10], train_loss : 2.0000 , val_loss : 1.7154, accuracy: 0.4106
Epoch [6/10], train_loss : 2.0000 , val_loss : 1.7232, accuracy: 0.4144
Epoch [7/10], train_loss : 2.0000 , val_loss : 1.7125, accuracy: 0.4295
Epoch [8/10], train_loss : 1.0000 , val_loss : 1.7372, accuracy: 0.4343
Epoch [9/10], train_loss : 1.0000 , val_loss : 1.6871, accuracy: 0.4441
Epoch [10/10], train_loss : 1.0000 , val_loss : 1.7384, accuracy: 0.4552
</code></pre>
<p>Using Pytorch is not converging.
I used the same network (Wideresnet22) and both are trained from scratch without pretrained model.</p>
<p>The network is <a href="https://www.dropbox.com/s/xuv0uyauw85ahzu/wideresnet22.py?dl=0" rel="nofollow noreferrer">here</a>.</p>
<p>Training using Pytorch is <a href="https://www.dropbox.com/s/wp2fsfrrt5f6u82/training_using_pytorch.py?dl=0" rel="nofollow noreferrer">here</a>.</p>
<p>Using Fastai is as follows.</p>
<pre><code>from fastai.basic_data import DataBunch
from fastai.train import Learner
from fastai.metrics import accuracy

#DataBunch takes data and internall create data loader
data = DataBunch.create(train_ds, valid_ds, bs=batch_size, path='./data')
#Learner uses Adam as default for learning
learner = Learner(data, model, loss_func=F.cross_entropy, metrics=[accuracy])
#Gradient is clipped
learner.clip = 0.1

#learner finds its learning rate
learner.lr_find()

learner.recorder.plot()

#Weight decay helps to lower down weight. Learn in https://towardsdatascience.com/
learner.fit_one_cycle(5, 5e-3, wd=1e-4)
</code></pre>
<p>What could be wrong in my training algorithm using Pytorch?</p>