title: Increase or decrease learning rate for adding neurons or weights?
body: <p>I have a convolutional neural network of which I modified the architecture. I do not have time to retrain and perform a cross-validation (grid search over optimal parameters). I want to <em>intuitively adjust</em> the learning rate.</p>
  
  <p>Should I <em>increase</em> or <em>decrease</em> the learning rate of my RMS (SGD-based) optimiser if: </p>
  
  <ol>
  <li>I add <em>more</em> neurons to the fully connected layers?</li>
  <li>on a convolutional neural network, I remove a sub-sampling (average or max pooling) layer before the full connections, and I increase the amount of fully connected units between that feature map and the softmax outputs (so that there are <em>more</em> weights connected to the fully connected neurons on top)?</li>
  </ol>