Choosing Number of Factors
When creating a regression model, it is critical to choose the number of factors that best represents the correlation of information in the data block to the property of interest. There are many methods that suggest looking at the shape of–
Choosing Number of Factors
When creating a regression model, it is critical to choose the number of factors that best represents the correlation of information in the data block to the property of interest. There are many methods that suggest looking at the shape of–or values in–a plot of eigenvalues: where the shape of the curve bends, like in a hockey stick; or stopping when the eigenvalues drop below 1; or simply assuring that 95% (or other subjective cutoff) of the variance is explained. But these methods are looking only at variance and not necessarily correlation to the Y value. And, they pertain only to modeling and not prediction.
We could look at errors following a prediction on the calibration data: as the PRESS–the prediction residual error sum of squares–or as the RMSEC, the root mean square error of calibration–correcting PRESS for the number of samples and the number of factors in the model. However, this approach is biased and optimistic because the error diagnostic is derived from data used to make the model in the first place. Ideally, we want to evaluate the model size based on how well it predicts a set of unknowns. But, which unknowns?
One way to simplify this selection is to use cross-validation. In this procedure, one or more samples are excluded from the data set, a model is made on the remaining data, then this model is used to predict the excluded samples. Error diagnostics from this prediction are saved, then different samples are excluded and the process repeated. After all samples have been removed once, a summary of the prediction errors can be shown as the root mean square error of cross validation or RMSECV. This diagnostic is less biased than the RMSEC.
Ideally, the shape of the RMSECV curve, as a function of number of factors, will rise when too many factors are chosen because such a model has incorporated noise which cannot be predicted. Such a model is overfitting the data. Thus, choosing the minimum in this curve is a good choice for number of factors.
It is often the case that the RMSECV for a model with fewer factors than that at the minimum of the curve is not statistically different. In general we recommend using the more parsimonious model, which further reduces the risk of overfitting. In addition to looking at the RMSECV, you should also look at other computed objects, such as the loadings, the regression vector and the spectral residuals.
For spectroscopic data, neighboring values of the X data are highly correlated such that the curve is quite smooth. Similarly, the regression vector should be smooth. When too many factors are retained in the model, the regression vector will begin to show a jagged appearance, indicating that noise is being added to the model. Another diagnostic, simply called the jaggedness, attempts to characterize this tendency of the regression vector to become noisy when the model is overfit; it will usually have a U shape where the minimum would be the suggested model optimum. This metric seems to more consistently suggest the optimal number of factors and we n