488
CHAPTER 12. LARGE-SCALE MACHINE LEARNING
w.x + b = +1
γ
γ
w.x + b = 0
w.x
+ b = 0
+ b = −1
w
||
/ || w
1
2
x
x
Figure 12.15: Normalizing the weight vector for an SVM
Regrouping terms, we see
w.x2 + b + 2γ w.w
∥w∥= 1
(12.2)
But the ﬁrst two terms of Equation 12.2, w.x2 + b, sum to −1, since we know
that x2 is on the lower hyperplane, w.x + b = −1. If we move this −1 from left
to right in Equation 12.2 and then divide through by 2, we conclude that
γ w.w
∥w∥= 1
(12.3)
Notice also that w.w is the sum of the squares of the components of w.
That is, w.w = ∥w∥2. We conclude from Equation 12.3 that γ = 1/∥w∥.
This equivalence gives us a way to reformulate the optimization problem
originally stated in Section 12.3.1. Instead of maximizing γ, we want to mini-
mize ∥w∥, which is the inverse of γ. That is:
• Given a training set (x1, y1), (x2, y2), . . . , (xn, yn), minimize ∥w∥(by vary-
ing w and b) subject to the constraint that for all i = 1, 2, . . . , n,
yi(w.xi + b) ≥1
Example 12.8 : Let us consider the four points of Fig. 11.1, supposing that
they alternate as positive and negative examples.
That is, the training set
consists of
([1, 2], +1)
([2, 1], −1)
([3, 4], +1)
([4, 3], −1)
