are three measurements per subject and treating them as independent observations would cause information to be leaked into each training/bag set from each validation set or out-of-bag set. So, long as all three measurements per subject remain together, bootstrapping and k-fold cross-validation can occur.

However, the typical 5-fold cross-validation reduces the training set size by 20% and bootstrapping reduces its information content by 36% on average. Both of these reductions are too much for data this small, and cause the performance of all models to drop to insignificant levels. As such, we also use a Leave-One-Subject-Out jackknife estimate for the standard error of accuracy, but this precludes estimating the standard error for the F1 score, and the MCC.

Any transformation of the data must be done within each fold of the jackknife to prevent any information leakage. This includes not just the training of the models but under- or over-sampling, feature selection, or hyperparameter tuning. As we are focused on the feature selection, we did not implement any data balancing like in [@smote].

Prediction
----------

Instead of treating each observation independently, the three measurements per subject are aggregated for each prediction. The three measurements are each classified, and the more 