<h2 dir="auto">🐛 Bug</h2>
<p dir="auto">MultiStepLR drops learning rate by gamma**2 times instead of gamma on each milestone. After one epoch it restores the lr to normal value.</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<p dir="auto">Use the following code:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch
params = [torch.nn.Parameter(torch.randn(1, 1))]
optimizer = torch.optim.SGD(params, lr=0.2)
scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[2,8], gamma=0.1)
for epoch in range(10):
    print(scheduler.get_lr())
    scheduler.step()"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-s1">params</span> <span class="pl-c1">=</span> [<span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">Parameter</span>(<span class="pl-s1">torch</span>.<span class="pl-en">randn</span>(<span class="pl-c1">1</span>, <span class="pl-c1">1</span>))]
<span class="pl-s1">optimizer</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">optim</span>.<span class="pl-v">SGD</span>(<span class="pl-s1">params</span>, <span class="pl-s1">lr</span><span class="pl-c1">=</span><span class="pl-c1">0.2</span>)
<span class="pl-s1">scheduler</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">optim</span>.<span class="pl-s1">lr_scheduler</span>.<span class="pl-v">MultiStepLR</span>(<span class="pl-s1">optimizer</span>, <span class="pl-s1">milestones</span><span class="pl-c1">=</span>[<span class="pl-c1">2</span>,<span class="pl-c1">8</span>], <span class="pl-s1">gamma</span><span class="pl-c1">=</span><span class="pl-c1">0.1</span>)
<span class="pl-k">for</span> <span class="pl-s1">epoch</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">10</span>):
    <span class="pl-en">print</span>(<span class="pl-s1">scheduler</span>.<span class="pl-en">get_lr</span>())
    <span class="pl-s1">scheduler</span>.<span class="pl-en">step</span>()</pre></div>
<p dir="auto">The output is:<br>
[0.2]<br>
[0.2]<br>
[0.0020000000000000005] here should be 0.02<br>
[0.020000000000000004]<br>
[0.020000000000000004]<br>
[0.020000000000000004]<br>
[0.020000000000000004]<br>
[0.020000000000000004]<br>
[0.00020000000000000006] here should be 0.002<br>
[0.0020000000000000005]</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">In the provided sample the learning rate should be adjusted only twice.</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">PyTorch version: 1.1.0<br>
Is debug build: No<br>
CUDA used to build PyTorch: 10.0.130</p>
<p dir="auto">OS: Ubuntu 16.04.6 LTS<br>
GCC version: (Ubuntu 5.5.0-12ubuntu1~16.04) 5.5.0 20171010<br>
CMake version: version 3.9.1</p>
<p dir="auto">Python version: 3.5<br>
Is CUDA available: Yes<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration:<br>
GPU 0: GeForce RTX 2080 Ti<br>
GPU 1: GeForce GTX 1080 Ti<br>
GPU 2: GeForce GTX 1080 Ti</p>
<p dir="auto">Nvidia driver version: 418.56<br>
cuDNN version: Could not collect</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip3] numpy==1.16.3<br>
[pip3] torch==1.1.0<br>
[pip3] torchstat==0.0.7<br>
[pip3] torchvision==0.2.2.post3<br>
[conda] Could not collect</p>
<p dir="auto">When the StepLR, MultiStepLR, ExponentialLR or CosineAnnealingLR scheduler is called with the same epoch parameter the optimizer value is further reduced even though it's the same epoch</p>
<p dir="auto">a sample code</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch.optim as optim
from torch import nn

conv = nn.Conv2d(3,3,3)
optimizer = optim.Adam(conv.parameters()) 
lr_scheduler = optim.lr_scheduler.StepLR(optimizer, 2)"><pre class="notranslate"><code class="notranslate">import torch.optim as optim
from torch import nn

conv = nn.Conv2d(3,3,3)
optimizer = optim.Adam(conv.parameters()) 
lr_scheduler = optim.lr_scheduler.StepLR(optimizer, 2)
</code></pre></div>
<p dir="auto">now if we call <code class="notranslate">lr_scheduler.step(epoch=2)</code> multiple times</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="for _ in range(10):
    lr_scheduler.step(2)
    print(optimizer.param_groups[0]['lr'])"><pre class="notranslate"><code class="notranslate">for _ in range(10):
    lr_scheduler.step(2)
    print(optimizer.param_groups[0]['lr'])
</code></pre></div>
<p dir="auto">output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; 0.0001
&gt;&gt;&gt; 1e-05
&gt;&gt;&gt; 1.0000000000000002e-06
&gt;&gt;&gt; 1.0000000000000002e-07
&gt;&gt;&gt; 1.0000000000000004e-08
&gt;&gt;&gt; 1.0000000000000005e-09
&gt;&gt;&gt; 1.0000000000000006e-10
&gt;&gt;&gt; 1.0000000000000006e-11
&gt;&gt;&gt; 1.0000000000000006e-12
&gt;&gt;&gt; 1.0000000000000007e-13"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; 0.0001
&gt;&gt;&gt; 1e-05
&gt;&gt;&gt; 1.0000000000000002e-06
&gt;&gt;&gt; 1.0000000000000002e-07
&gt;&gt;&gt; 1.0000000000000004e-08
&gt;&gt;&gt; 1.0000000000000005e-09
&gt;&gt;&gt; 1.0000000000000006e-10
&gt;&gt;&gt; 1.0000000000000006e-11
&gt;&gt;&gt; 1.0000000000000006e-12
&gt;&gt;&gt; 1.0000000000000007e-13
</code></pre></div>
<p dir="auto">Even if such use-case is bizarre, this is extremely unexpected.<br>
This is happens using PyTorch version 1.1.0 but not 1.0.1. Because pull(<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="380976590" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/14010" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/14010/hovercard" href="https://github.com/pytorch/pytorch/pull/14010">#14010</a>) redefined StepLR, MultiStepLR, ExponentialLR and CosineAnnealingLR to directly use the learning rate variable of the optimizer rather than using <code class="notranslate">base_lr</code> defined by <code class="notranslate">_LRScheduler</code>. This was in order to support multiple simultaneous schedulers (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="373244157" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/13022" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/13022/hovercard" href="https://github.com/pytorch/pytorch/issues/13022">#13022</a>)!</p>