518
CHAPTER 12. LARGE-SCALE MACHINE LEARNING
hyperplane. Since this vector is often of very high dimension, it can be very
hard to interpret the model.
Nearest-Neighbor Classiﬁcation and Regression: Here, the model is the
training set itself, so we expect it to be intuitively understandable. The ap-
proach can deal with multidimensional data, although the larger the number of
dimensions, the sparser the training set will be, and therefore the less likely it
is that we shall ﬁnd a training point very close to the point we need to classify.
That is, the “curse of dimensionality” makes nearest-neighbor methods ques-
tionable in high dimensions. These methods are really only useful for numerical
features, although one could allow categorical features with a small number of
values.
For instance, a binary categorical feature like {male, female} could
have the values replaced by 0 and 1, so there was no distance in this dimension
between individuals of the same gender and distance 1 between other pairs of
individuals. However, three or more values cannot be assigned numbers that are
equidistant. Finally, nearest-neighbor methods have many parameters to set,
including the distance measure we use (e.g., cosine or Euclidean), the number of
neighbors to choose, and the kernel function to use. Diﬀerent choices result in
diﬀerent classiﬁcation, and in many cases it is not obvious which choices yield
the best results.
Decision Trees: Unlike the other methods discussed in this chapter, Decision
trees are useful for both categorical and numerical features. The models pro-
duced are generally quite understandable, since each decision is represented by
one node of the tree. However, this approach is most useful for low-dimension
feature vectors. As discussed in Section 12.5.7, building decision trees with
many levels often leads to overﬁtting. But if a decision tree has few levels, then
it cannot even mention more than a small number of features. As a result, the
best use of decision trees is often to create a decision forest of many, low-depth
trees and combine their decision in some way.
12.7
Summary of Chapter 12
✦Training Sets: A training set consists of a feature vector, each component
of which is a feature, and a label indicating the class to which the object
represented by the feature vector belongs. Features can be categorical –
belonging to an enumerated list of values – or numerical.
✦Test Sets and Overﬁtting: When training some classiﬁer on a training set,
it is useful to remove some of the training set and use the removed data
as a test set. After producing a model or classiﬁer without using the test
set, we can run the classiﬁer on the test set to see how well it does. If the
classiﬁer does not perform as well on the test set as on the training set
used, then we have overﬁt the training set by conforming to peculiarities
of the training-set data which is not present in the data as a whole.
