490
CHAPTER 12. LARGE-SCALE MACHINE LEARNING
w.x + b = −1
w.x + b = +1
w.x + b = 0
Misclassified
Too close
to boundary
Figure 12.16: Points that are misclassiﬁed or are too close to the separating
hyperplane incur a penalty; the amount of the penalty is proportional to the
length of the arrow leading to that point
To see why minimizing the term ∥w∥2/2 makes sense, note that minimizing
∥w∥is the same as minimizing any monotone function of ∥w∥, so it is at least an
option to choose a formula in which we try to minimize ∥w∥2/2. This expression
is desirable because its derivative with respect to any component of w is that
component. That is, if w = [w1, w2, . . . , wd], then ∥w∥2/2 is 1
2
Pn
i=1 w2
i , so its
partial derivative ∂/∂wi is wi. This situation makes sense because, as we shall
see, the derivative of the penalty term with respect to wi is a constant times each
xi, the corresponding component of each feature vector whose training example
incurs a penalty. That in turn means that the vector w and the vectors of the
training set are commensurate in the units of their components.
Thus, we shall consider how to minimize the particular function
f(w, b) = 1
2
d
X
j=1
w2
j + C
n
X
i=1
max
n
0,
1 −yi
 d
X
j=1
wjxij + b
o
(12.4)
The ﬁrst term encourages small ∥w∥, while the second term, involving the
constant C that must be chosen properly, represents the penalty for bad points
in a manner to be explained below. We assume there are n training examples
(xi, yi) for i = 1, 2, . . ., n, and xi = [xi1, xi2, . . . , xid]. Also, as before, w =
[w1, w2, . . . , wd]. Note that the two summations Pd
j=1 express the dot product
of vectors.
The constant C, called the regularization parameter, reﬂects how important
misclassiﬁcation is.
Pick a large C if you really do not want to misclassify
