<p>There are actually two strategies for warmup, ref <a href="https://arxiv.org/abs/1706.02677" rel="nofollow noreferrer">here</a>.</p>
<ul>
<li><strong>constant</strong>: Use a low learning rate than base learning rate for the initial few steps.</li>
<li><strong>gradual</strong>: In the first few steps, the learning rate is set to be lower than base learning rate and increased gradually to approach it as step number increases. As @Prune an @Patel suggested.</li>
</ul>
What does "learning rate warm-up" mean?
<p>In machine learning, especially deep learning, what does it mean to warm-up?</p>
<p>I've heard sometimes that in some models, warming-up is a phase in training. But honestly, I don't know what it is because I'm very new to ML. Until now I've never used or come across it, but I want to know it because I think it might be useful for me.</p>
<p><strong>What is learning rate warm-up and when do we need it?</strong></p>