class $F$ of functions mapping the input space $X$ to the output space $Y$, the ultimate goal is to find a model $f \in F$ which is a good approximation of the labels actually observed in the data. Given a labeled training data set $D$ and a cost function $l$, the process goes as follows: the cost function assigns a numerical value to each combination of data point, true label, and classifier label. The classifier then tries to “learn” the function that minimizes the cost incurred in predicting the label $y_i$ of a new data point $(x_i, y_i)$ when only the predictor variable $x_i$ is revealed to the classifier.

In simple terms, if there is some “true” function $h$ we are trying to learn, the goal is to find a function $f \in F$ which is as “close” to $h$ as possible, given the constraints that the model class imposes on us. When the labels are real numbers ($Y=\mathbb{R}$), the paradigm is called regression learning, and when $Y$ is a finite set of labels it is called classification learning.

In regression learning, since the label values are real numbers, a loss function is used to penalize predictions that are made 