<p>It is a common practice to decrease the learning rate (lr) as the optimization/learning process progresses. However, it is not clear how exactly the learning rate should be decreased as a function of the iteration number.</p>

<p>If you use <a href="https://github.com/NVIDIA/DIGITS" rel="noreferrer">DIGITS</a> as an interface to Caffe, you will be able to visually see how the different choices affect the learning rate.</p>

<p><strong>fixed:</strong> the learning rate is kept fixed throughout the learning process.</p>

<hr>

<p><strong>inv:</strong> the learning rate is decaying as ~<code>1/T</code><br>
<img src="https://i.stack.imgur.com/LScLY.png" alt="enter image description here"></p>

<hr>

<p><strong>step:</strong> the learning rate is piecewise constant, dropping every X iterations<br>
<img src="https://i.stack.imgur.com/W5h6j.png" alt="enter image description here"> </p>

<hr>

<p><strong>multistep:</strong> piecewise constant at arbitrary intervals<br>
<img src="https://i.stack.imgur.com/DW0qa.png" alt="enter image description here"></p>

<hr>

<p>You can see exactly how the learning rate is computed in the function <a href="https://github.com/BVLC/caffe/blob/master/src/caffe/solvers/sgd_solver.cpp#L27" rel="noreferrer"><code>SGDSolver&lt;Dtype&gt;::GetLearningRate</code></a> (<em>solvers/sgd_solver.cpp</em> line ~30).</p>

<hr>

<p>Recently, I came across an interesting and unconventional approach to learning-rate tuning: <a href="http://arxiv.org/abs/1506.01186" rel="noreferrer">Leslie N. Smith's work "No More Pesky Learning Rate Guessing Games"</a>. In his report, Leslie suggests to use <code>lr_policy</code> that alternates between decreasing and <em>increasing</em> the learning rate. His work also suggests how to implement this policy in Caffe.</p>
What is `lr_policy` in Caffe?
<p>I just try to find out how I can use <a href="http://caffe.berkeleyvision.org/">Caffe</a>. To do so, I just took a look at the different <code>.prototxt</code> files in the examples folder. There is one option I don't understand:</p>

<pre><code># The learning rate policy
lr_policy: "inv"
</code></pre>

<p>Possible values seem to be:</p>

<ul>
<li><code>"fixed"</code></li>
<li><code>"inv"</code></li>
<li><code>"step"</code></li>
<li><code>"multistep"</code></li>
<li><code>"stepearly"</code></li>
<li><code>"poly"</code>  </li>
</ul>

<p>Could somebody please explain those options?</p>