<p>There are quite a lot of possible options for you to tune here; what you are currently facing is generally referred to as "Hyperparameter Optimization", which is an entire research field by itself.</p>

<p>Basically, you want to tune your parameters in such a way that you get the best possible result. First of all, I would recommend you to simply train for a longer period of time (more epochs).
I have never trained on CIFAR-10 myself, but it could be that convergence is reached much later (although I doubt it). Also, I would recommend you to implement something along the lines of early stopping.</p>

<p>Instead of always using the latest model, instead use (i.e. checkpoint) the one with the highest validation accuracy, in the hopes that this is the one that generalizes best to unseen data. Even though it might not have perfect scores on your training data, it will most of the times serve you better in practice.<br/>
After completing your whole run (or without significant improvement in terms of validation loss), you can then cancel your training procedure early.</p>

<p>Furthermore, I am not sure how much you have played around with the internals of the residual blocks, but maybe you can try to adjust some parameters here, like the convolutional size, convolutions per residual block, number of features, etc. etc.</p>

<p>There is really a lot to look out for, but I am assuming you are mostly sticking to the <a href="https://arxiv.org/abs/1512.03385" rel="nofollow noreferrer">ResNet paper</a> in terms of your architecture.
Sadly, there is also a good deal of luck involved for that, since a good random initialization might yield far better results with otherwise similar properties (see <a href="https://arxiv.org/abs/1803.03635" rel="nofollow noreferrer">this paper</a> for more details), so simply choosing a different random seed might help out...</p>

<p>Last, but not least, I would recommend you to look into different optimizers: Generally, default SGD "only works so well" (although it is still a great tool and might in some cases work best). Adding more advanced metrics, with the simplest one being for example <a href="https://towardsdatascience.com/stochastic-gradient-descent-with-momentum-a84097641a5d" rel="nofollow noreferrer"><em>momentum</em></a>, may result in better convergence.</p>

<p>Most of the deep learning tools out there offer you a wide variety of optimizers. A decent go-to choice would be the Adam optimizer. There are of course different ones out there, an overview can be found <a href="http://,%20or%20variations%20of%20AdaGrad,%20that%20use%20%22some%20fancy%20math%22%20(specifically,%20derivative%20information,%20or" rel="nofollow noreferrer">here</a>.</p>
Tensorflow: Learning rate got saturated in resnet-56 after 50 epoches training, but test accuracy is still not satisfactory
<p>I built a 56 layer residual network to train the CIFAR-10 dataset for image classification. Though it's the state-of-the-art network architecture, I get my model test accuracy as 79% at 10 epoches training.</p>

<p>The training dataset size is 49000, and the validation dataset size is 1000. I trained the model in 20 epoches and the minibatch size as 128. The learning rate is 1e-3. I used Xavier initialization and RMProp for gradient descent.</p>

<p>Refer here for my implementation. 
<a href="https://github.com/Jiancong/cs231n_2017/blob/master/assignment2/TensorFlow.ipynb" rel="nofollow noreferrer">https://github.com/Jiancong/cs231n_2017/blob/master/assignment2/TensorFlow.ipynb</a></p>

<p>The result is following
<a href="https://i.stack.imgur.com/Ggu7C.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Ggu7C.png" alt="enter image description here"></a></p>

<p>I tried to decrease the learning rate to 1e-4, meanwhile the test accuracy degraded too. And I tried to increase the training epoches to 15. The accuracy increased as following.</p>

<p><a href="https://i.stack.imgur.com/4flxl.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4flxl.png" alt="enter image description here"></a></p>

<p>I increased the training epoches to 50, and test accuracy got saturated at 85%. </p>

<p>Increased training epoches seems not work now. Anything I worthy to try?</p>