486
CHAPTER 12. LARGE-SCALE MACHINE LEARNING
12.3.1
The Mechanics of an SVM
The goal of an SVM is to select a hyperplane w.x + b = 0 that maximizes
the distance γ between the hyperplane and any point of the training set.1 The
idea is suggested by Fig. 12.14. There, we see the points of two classes and a
hyperplane dividing them.
Support
vectors
γ
γ
w.x + b = 0
Figure 12.14: An SVM selects the hyperplane with the greatest possible margin
γ between the hyperplane and the training points
Intuitively, we are more certain of the class of points that are far from the
separating hyperplane than we are of points near to that hyperplane. Thus, it
is desirable that all the training points be as far from the hyperplane as possible
(but on the correct side of that hyperplane, of course). An added advantage
of choosing the separating hyperplane to have as large a margin as possible is
that there may be points closer to the hyperplane in the full data set but not
in the training set. If so, we have a better chance that these points will be
classiﬁed properly than if we chose a hyperplane that separated the training
points but allowed some points to be very close to the hyperplane itself. In that
case, there is a fair chance that a new point that was near a training point that
was also near the hyperplane would be misclassiﬁed. This issue was discussed
in Section 12.2.7 in connection with Fig. 12.13.
We also see in Fig. 12.14 two parallel hyperplanes at distance γ from the
central hyperplane w.x + b = 0, and these each touch one or more of the
support vectors. The latter are the points that actually constrain the dividing
hyperplane, in the sense that they are all at distance γ from the hyperplane.
In most cases, a d-dimensional set of points has d + 1 support vectors, as is
the case in Fig. 12.14. However, there can be more support vectors if too many
points happen to lie on the parallel hyperplanes. We shall see an example based
1Constant b in this formulation of a hyperplane is the same as the negative of the threshold
θ in our treatment of perceptrons in Section 12.2.
