466
CHAPTER 12. LARGE-SCALE MACHINE LEARNING
(2,1)
(3,4)
(4,3)
(1,2)
Figure 12.2: Repeat of Fig. 11.1, to be used as a training set
of “best” is that the RMSE of the value of f(x) compared with the given value
of y is minimized. That is, we want to minimize
4
X
x=1
(ax + b −yx)2
where yx is the y-value associated with x. This sum is
(a + b −2)2 + (2a + b −1)2 + (3a + b −4)2 + (4a + b −3)2
Simplifying, the sum is 30a2 + 4b2 + 20ab −56a −20b + 30. If we then take the
derivatives with respect to a and b and set them to 0, we get
60a + 20b −56
=
0
20a + 8b −20
=
0
The solution to these equations is a = 3/5 and b = 1. For these values the
RMSE is 3.2.
Note that the learned straight line is not the principal axis that was dis-
covered for these points in Section 11.2.1. That axis was the line with slope
1, going through the origin, i.e., the line y = x. For this line, the RMSE is
4. The diﬀerence is that PCA discussed in Section 11.2.1 minimizes the sum
of the squares of the lengths of the projections onto the chosen axis, which is
constrained to go through the origin. Here, we are minimizing the sum of the
squares of the vertical distances between the points and the line. In fact, even
had we tried to learn the line through the origin with the least RMSE, we would
not choose the line y = x. You can check that y = 14
15x has a lower RMSE than
4.
✷
Example 12.3 : A common application of machine learning involves a training
set where the feature vectors x are Boolean-valued and of very high dimension.
We shall focus on data consisting of documents, e.g., emails, Web pages, or
newspaper articles. Each component represents a word in some large dictio-
nary. We would probably eliminate stop words (very common words) from this
