title: Momentum in neural networks
body: <h1>Neural networks and momentum</h1>
  
  <p>Should the momentum factor preferably relate to [both the dataset instance and the individual weights] or [just the weights]. Eg:</p>
  
  <pre><code>def get_momentum( instance, weight ):
     return float
  
  instance1 = vector 1xn
  instance2 = vector 1xn
  weights   = vector 1xn
  
  # Option 1
  get_momentum( instance1, weights[0] ) # eg returns 0.1
  get_momentum( instance2, weights[0] ) # eg returns 0.3 &lt;-- same weight, different momentum
  
  # Option 2
  get_momentum( instance1, weights[0] ) # eg returns 0.1
  get_momentum( instance2, weights[0] ) # eg returns 0.1
  </code></pre>
  
  <p>The second alternative would have lower memory complexity. I believe it would also cause the learning algorithm to be more likely to get stuck in local optima than the first alternative. Option 1 should cause a stronger momentum "pull".  </p>