title: classification of machine learning optimization algorithms
body: <p>What are the various categories of machine learning optimization algorithms?
  At some place its mentioned that Optimization Algorithm falls under 2 major categories -</p>
  
  <pre><code>    First Order Optimization Algorithms and second order Optimization Algorithms
  </code></pre>
  
  <p>Some also classify it majorly as-</p>
  
  <pre><code>    Convex Optimization and Non-Convex Optimization
  </code></pre>
  
  <p>Whereas others also view it as-</p>
  
  <pre><code>    Constrained and Unconstrained Optimization
  </code></pre>
  
  <p>After all these classifications I cant seem to understand what falls under what category and please do point me towards any classification tree for the same if it exists.</p>