522
CHAPTER 13. NEURAL NETS AND DEEP LEARNING
the observed output corresponding to input x is ˆy and the predicted output is
y. Then a loss fucntion L(y, ˆy) quantiﬁes the prediction error for this single
input. Typically, we consider the loss over a large set of observations, such
as the entire training set. In that case, we usually average the losses over all
training examples.
We shall consider separately two cases. In the ﬁrst case, there is a single
output node, and it produces a real value. In this case we study “regression
loss.” In the second case, there are several output nodes, each of which indicates
that the input is a member of a particular class; we study this matter under
“classiﬁcation loss” in Section 13.2.9.
13.2.8
Regression Loss
Suppose the model has a single continuous-valued output, and (x, ˆy) is a training
example. For the same input x, suppose the predicted output of the neural net
is y. Then the squared error loss L(y, ˆy) of this prediction is:
L(y, ˆy) = (y −ˆy)2
In general, we compute the loss for a set of predictions. Suppose the observed
(i.e., training set) input-output pairs are T = {(x1, ˆy1), (x2, ˆy2), . . . , (xn, ˆyn)},
while the corresponding input-output pairs predicted by the model are P =
{(x1, y1), (x2, y2), . . . , (xn, yn)}. The mean squared error (MSE) for the set is:
L(P, T ) = 1
n
n
X
i=1
(yi −ˆyi)2
Note that the mean squared error is just square of the RMSE. It is convenient
to omit the square root to simplify the derivative of the function, which we shall
use during training. In any case, when we minimize MSE we also automatically
minimize RMSE.
One problem with MSE is that it is very sensitive to outliers due the squared
term. A few outliers can contribute very highly to the loss and swamp out the
eﬀect of other points, making the training process susceptible to wild swings.
One way to deal with this issue is to use the Huber Loss. Suppose z = y −ˆy,
and δ is a constant. The Huber Loss Lδ is given by:
Lδ(z) =
(
z2
if |z| ≤δ
2δ(|z| −1
2δ)
otherwise
Figure 13.6 contrasts the squared error and Huber loss functions.
In the case where we have a vector y of outputs rather than a single output,
we use ∥y −ˆy∥in place of |y −ˆy| in the deﬁnitions of mean squared error and
Huber loss.
