12.3. SUPPORT-VECTOR MACHINES
491
points, but you would accept a narrow margin. Pick a small C if you are OK
with some misclassiﬁed points, but want most of the points to be far away from
the boundary (i.e., the margin is large).
We must explain the penalty function (second term) in Equation 12.4. The
summation over i has one term
L(xi, yi) = max
n
0,
1 −yi
 d
X
j=1
wjxij + b
o
for each training example xi. L is the hinge function, suggested in Fig. 12.17,
and we call its value the hinge loss. Let zi = yi(Pd
j=1 wjxij + b). When zi is 1
or more, the value of L is 0. But for smaller values of zi, L rises linearly as zi
decreases.
−2
−1
0
1
2
3
0
1
2
max{0, 1−z }
z = yi
w .
ix
(
+ b )
Figure 12.17: The hinge function decreases linearly for z ≤1 and then remains
0
Since we shall have need to take the derivative with respect to each wj of
L(xi, yi), note that the derivative of the hinge function is discontinuous. It is
−yixij for zi < 1 and 0 for zi > 1. That is, if yi = +1 (i.e., the ith training
example is positive), then
∂L
∂wj
= if
d
X
j=1
wjxij + b ≥1 then 0 else −xij
Moreover, if yi = −1 (i.e., the ith training example is negative), then
∂L
∂wj
= if
d
X
j=1
wjxij + b ≤−1 then 0 else xij
The two cases can be summarized as one, if we involve the value of yi, as:
∂L
∂wj
= if yi(
d
X
j=1
wjxij + b) ≥1 then 0 else −yixij
(12.5)
