<p>As a short and <strong>practical</strong> answer, here the learning rate is decreased if the model is more complex, the variable <code>model_size</code> is approximately the number of neurons per layer: </p>

<pre><code>def rate(self, step = None):
    "Implement `lrate` above"
    if step is None:
        step = self._step
    return self.factor * \
        (self.model_size ** (-0.5) *
        min(step ** (-0.5), step * self.warmup ** (-1.5)))
</code></pre>

<p>Source: <a href="https://github.com/harvardnlp/annotated-transformer/blob/master/The%20Annotated%20Transformer.ipynb" rel="nofollow noreferrer">The Annotated Transformer</a></p>

<p>Also see: <a href="https://arxiv.org/abs/1412.6980" rel="nofollow noreferrer">Adam: A Method for Stochastic Optimization</a></p>
Increase or decrease learning rate for adding neurons or weights?
<p>I have a convolutional neural network of which I modified the architecture. I do not have time to retrain and perform a cross-validation (grid search over optimal parameters). I want to <em>intuitively adjust</em> the learning rate.</p>

<p>Should I <em>increase</em> or <em>decrease</em> the learning rate of my RMS (SGD-based) optimiser if: </p>

<ol>
<li>I add <em>more</em> neurons to the fully connected layers?</li>
<li>on a convolutional neural network, I remove a sub-sampling (average or max pooling) layer before the full connections, and I increase the amount of fully connected units between that feature map and the softmax outputs (so that there are <em>more</em> weights connected to the fully connected neurons on top)?</li>
</ol>