<p>We all agree that the learning rate can be seen as a way to control overfitting, just like dropout or batch size. But I'm writing this answer because I think the following in Amir's answer and comments is misleading :</p>

<ul>
<li><blockquote>
  <p>adding more layers/neurons increases the chance of over-fitting.
  Therefore it would be better if you decrease the learning rate over
  time.</p>
</blockquote></li>
<li><blockquote>
  <p>Since adding more layers/nodes to the model makes it prone to
  over-fitting [...]
  taking small steps towards the local minima is recommended</p>
</blockquote></li>
</ul>

<p>It's actually the <strong>OPPOSITE</strong>! A smaller learning rate will <em>increase</em> the risk of overfitting!</p>

<p>Citing from <a href="https://arxiv.org/pdf/1708.07120.pdf" rel="noreferrer">Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates (Smith &amp; Topin 2018)</a> (a very interesting read btw):</p>

<blockquote>
  <p>There are many forms of regularization, such as <strong>large learning rates</strong>,
  small batch sizes, weight decay, and dropout. Practitioners must
  balance the various forms of regularization for each dataset and
  architecture in order to obtain good performance. <strong>Reducing other forms
  of regularization and regularizing with very large learning rates
  makes training significantly more efficient.</strong></p>
</blockquote>

<p>So, as Guillaume Chevalier said in his first comment, if you add regularization, decreasing the learning rate might be a good idea if you want to keep the overall amount of regularization constant. But if your goal is to increase the overall amount of regularization, or if you reduced other means of regularization (e.g., decreased dropout, increased batch size), then the learning rate should be <em>increased</em>.</p>

<p>Related (and also very interesting): <a href="https://arxiv.org/pdf/1711.00489.pdf" rel="noreferrer">Don't decay the learning rate, increase the batch size (Smith et al. ICLR'18)</a>.</p>
Increase or decrease learning rate for adding neurons or weights?
<p>I have a convolutional neural network of which I modified the architecture. I do not have time to retrain and perform a cross-validation (grid search over optimal parameters). I want to <em>intuitively adjust</em> the learning rate.</p>

<p>Should I <em>increase</em> or <em>decrease</em> the learning rate of my RMS (SGD-based) optimiser if: </p>

<ol>
<li>I add <em>more</em> neurons to the fully connected layers?</li>
<li>on a convolutional neural network, I remove a sub-sampling (average or max pooling) layer before the full connections, and I increase the amount of fully connected units between that feature map and the softmax outputs (so that there are <em>more</em> weights connected to the fully connected neurons on top)?</li>
</ol>