title: Solving nonlinear equations numerically
body: <p>I need to solve nonlinear minimization (least residual squares of N unknowns) problems in my Java program. The usual way to solve these is the <a href="http://en.wikipedia.org/wiki/Levenberg-Marquardt_algorithm" rel="nofollow noreferrer">Levenberg-Marquardt</a> algorithm. I have a couple of questions</p>
  
  <ul>
  <li><p>Does anybody have experience on the different LM implementations available? There exist slightly different flavors of LM, and I've heard that the exact implementation of the algorithm has a major effect on the its numerical stability. My functions are pretty well-behaved so this will probably not be a problem, but of course I'd like to choose one of the better alternatives. Here are some alternatives I've found:</p>
  
  <ul>
  <li><p><a href="http://www1.fpl.fs.fed.us/optimization.html" rel="nofollow noreferrer">FPL Statistics Group's Nonlinear Optimization Java Package</a>. This includes a Java translation of the classic Fortran MINPACK routines.</p></li>
  <li><p><a href="http://icl.cs.utk.edu/f2j/" rel="nofollow noreferrer">JLAPACK</a>, another Fortran translation.</p></li>
  <li><p><a href="http://optalgtoolkit.sourceforge.net/" rel="nofollow noreferrer">Optimization Algorithm Toolkit</a>.</p></li>
  <li><p><a href="http://scribblethink.org/Computer/Javanumeric/index.html" rel="nofollow noreferrer">Javanumerics</a>.</p></li>
  <li><p>Some Python implementation. Pure Python would be fine, since it can be compiled to Java with jythonc.</p></li>
  </ul></li>
  <li><p>Are there any commonly used heuristics to do the initial guess that LM requires?</p></li>
  <li><p>In my application I need to set some constraints on the solution, but luckily they are simple: I just require that the solutions (in order to be physical solutions) are nonnegative. Slightly negative solutions are result of measurement inaccuracies in the data, and should obviously be zero. I was thinking to use "regular" LM but iterate so that if some of the unknowns becomes negative, I set it to zero and resolve the rest from that. Real mathematicians will probably laugh at me, but do you think that this could work?</p></li>
  </ul>
  
  <p>Thanks for any opinions!</p>
  
  <p><strong>Update</strong>: This is not rocket science, the number of parameters to solve (N) is at most 5 and the data sets are barely big enough to make solving possible, so I believe Java is quite efficient enough to solve this. And I believe that this problem has been solved numerous times by clever applied mathematicians, so I'm just looking for some ready solution rather than cooking my own. E.g. Scipy.optimize.minpack.leastsq would probably be fine if it was pure Python.. </p>