<p>For SVM's, the goal is to find a classifier. This problem can be expressed in terms of a function that you are trying to minimize.
Let's first consider the <strong>Newton iteration</strong>. Newton iteration is a numerical method to find a solution to a problem of the form f(x) = 0.
Instead of solving it <strong>analytically</strong> we can solve it <strong>numerically</strong> by the follwing iteration:</p>

<pre><code>x^k+1 = x^k - DF(x)^-1 * F(x)
</code></pre>

<p>Here <code>x^k+1</code> is the k+1th iterate, <code>DF(x)^-1</code> is the inverse of the Jacobian of F(x) and <code>x</code> is the kth x in the iteration.</p>

<p>This update runs as long as we make progress in terms of step size (delta x) or if our function value approaches 0 to a good degree. The termination criteria can be chosen accordingly.</p>

<p>Now consider solving the problem <code>f'(x)=0</code>. If we formulate the Newton iteration for that, we get</p>

<pre><code>x^k+1 = x - HF(x)^-1 * DF(x)
</code></pre>

<p>Where <code>HF(x)^-1</code> is the inverse of the Hessian matrix and <code>DF(x)</code> the gradient of the function F. Note that we are talking about n-dimensional Analysis and can not just take the quotient. We have to take the inverse of the matrix.</p>

<p>Now we are facing some problems: In each step, we have to calculate the Hessian matrix for the updated x, which is very inefficient. We also have to solve a system of linear equations, namely <code>y = HF(x)^-1 * DF(x)</code> or <code>HF(x)*y = DF(x)</code>.
So instead of computing the Hessian in every iteration, we start off with an initial guess of the Hessian (maybe the identity matrix) and perform rank one updates after each iterate. For the exact formulas have a look <a href="https://en.wikipedia.org/wiki/Broyden%E2%80%93Fletcher%E2%80%93Goldfarb%E2%80%93Shanno_algorithm" rel="nofollow noreferrer">here</a>.</p>

<p>So how does this link to SVM's?</p>

<p>When you look at the function you are trying to minimize, you can formulate a primal problem, which you can the reformulate as a Dual Lagrangian problem which is convex and can be solved numerically. It is all well documented in the article so I will not try to express the formulas in a less good quality.</p>

<p>But the idea is the following: If you have a dual problem, you can solve it numerically. There are multiple solvers available. In the link you posted, they recommend coordinate descent, which solves the optimization problem for one coordinate at a time. Or you can use subgradient descent. Another method is to use L-BFGS. It is really well explained in <a href="https://arxiv.org/abs/1402.4861" rel="nofollow noreferrer">this</a> paper.</p>

<p>Another popular algorithm for solving problems like that is ADMM (alternating direction method of multipliers). In order to use ADMM you would have to reformulate the given problem into an equal problem that would give the same solution, but has the correct format for ADMM. For that I suggest reading Boyds script on ADMM.</p>

<p>In general: First, understand the function you are trying to minimize and then choose the numerical method that is most suited. In this case, subgradient descent and coordinate descent are most suited, as stated in the Wikipedia link.</p>
Quadratic Programming and quasi newton method BFGS
<p>Yesterday, I posted a question about general concept of SVM Primal Form Implementation:</p>

<p><a href="https://stackoverflow.com/questions/23752856/support-vector-machine-primal-form-implementation/23760112#23760112">Support Vector Machine Primal Form Implementation</a></p>

<p>and "lejlot" helped me out to understand that what I am solving is a QP problem.</p>

<p>But I still don't understand how my objective function can be expressed as QP problem</p>

<p>(<a href="http://en.wikipedia.org/wiki/Support_vector_machine#Primal_form" rel="nofollow noreferrer">http://en.wikipedia.org/wiki/Support_vector_machine#Primal_form</a>)</p>

<p>Also I don't understand how QP and Quasi-Newton method are related </p>

<p>All I know is Quasi-Newton method will SOLVE my QP problem which supposedly formulated from</p>

<p>my objective function (which I don't see the connection)</p>

<p>Can anyone walk me through this please??</p>