<p>You might get some use out of this thread: <a href="https://stackoverflow.com/questions/59996859/how-to-use-pytorch-onecyclelr-in-a-training-loop-and-optimizer-scheduler-intera">How to use Pytorch OneCycleLR in a training loop (and optimizer/scheduler interactions)?</a></p>
<p>But to address your points:</p>
<ol>
<li><p>Does the max_lr parameter has to be same with the optimizer lr parameter? <strong>No, this is the max or highest value -- a hyperparameter that you will experiment with. Notice in the paper the use of max_lr: <a href="https://arxiv.org/pdf/1708.07120.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1708.07120.pdf</a></strong></p>
</li>
<li><p>Can this scheduler be used with Adam optimizer. How is the momentum calculated then? <strong>Yes.</strong></p>
</li>
<li><p>Let’s say i trained my model for some number of epochs at a stretch now, i wanted to train for some more epochs. Would i have to reset the the scheduler?  <strong>Depends, are you loading the model from a saved checkpoint or not? Check PyTorch's tutorials: <a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#sphx-glr-beginner-blitz-neural-networks-tutorial-py" rel="nofollow noreferrer">https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#sphx-glr-beginner-blitz-neural-networks-tutorial-py</a></strong></p>
</li>
</ol>
How does one use torch.optim.lr_scheduler.OneCycleLR()?
<p>I wanted to use <code>torch.optim.lr_scheduler.OneCycleLR()</code> while training. Can some kindly explain to me how to use it?
What i got from the documentation was that it should be called after each train_batch.</p>
<p>My confusions are as follows:</p>
<ul>
<li><p>Does the max_lr parameter has to be same with the optimizer lr parameter?</p>
</li>
<li><p>Can this scheduler be used with Adam optimizer. How is the momentum calculated then?</p>
</li>
<li><p>Let’s say i trained my model for some number of epochs at a stretch now, i wanted to train for some more epochs. Would i have to reset the the scheduler?</p>
</li>
</ul>
<p>Can anybody provide me a sort of a toy example/training loop that implements this scheduler?</p>
<p>I am kind of new to deep learning &amp; PyTorch so my question might be somewhat silly.</p>