title: Need good way to choose and adjust a "learning rate"
body: <p>In the picture below you can see a learning algorithm trying to learn to produce a desired output (the red line).  The learning algorithm is similar to a backward error propagation neural network.</p>
  
  <p>The "learning rate" is a value that controls the size of the adjustments made during the training process.  If the learning rate is too high, then the algorithm learns quickly but its predictions jump around a lot during the training process (green line - learning rate of 0.001), if it is lower then the predictions jump around less, but the algorithm takes a lot longer to learn (blue line - learning rate of 0.0001).</p>
  
  <p>The black lines are moving averages.</p>
  
  <p>How can I adapt the learning rate so that it converges to close to the desired output initially, but then slows down so that it can hone in on the correct value?</p>
  
  <p><a href="http://img.skitch.com/20090605-pqpkse1yr1e5r869y6eehmpsym.png" rel="nofollow noreferrer">learning rate graph http://img.skitch.com/20090605-pqpkse1yr1e5r869y6eehmpsym.png</a></p>