Model Development and Validation
class-imbalance
REGRESSION
In this study, the ML-based prediction models were established using the following steps:Feature selection: we applied CatBoost and LightGBM to initially evaluate the importance of individual features. Moreover, least absolute shrinkage and selection operator (LASSO) analysis was used to select the key predictors. LASSO is a form of L1 regularization that adds the absolute magnitude of feature coefficients as the penalty term rather than the squared magnitude that is typically used [Algorithm selection and optimization: a total of 5 commonly used ML algorithms, including CatBoost, LightGBM, back propagation–neural network, random forest, and logistic regression, were applied to build the prediction models. We used the grid search strategy to identify the optimal combination of hyperparameters of these models to improve the prediction performance. Ten-fold stratified cross-validation was performed inside each grid option, and the optimal hyperparameter set was chosen based on the model in the grid search with the highest F1 score.Model fitting: before model fitting, a combination of the synthetic minority oversampling technique and undersampling was used to overcome the class-imbalance issue in this study. Qualified data from the MIMIC-IV database were randomly split into a training set (80% of the sample) and an internal validation set (20% of the sample) using the In this study, the prediction models were validated both internally and externally. The discriminatory ability of the prediction model was evaluated by using the area under the receiver operating characteristic curve (AUC). The calibrated ability was qualitatively and quantitively assessed by the calibration curve and the Brier score, respectively. Calibration reflects the extent to which the predicted probabilities and actual probabilities agree and is qualitatively and quantitatively evaluated through the calibration curve and Brier score, respectively. The Brier score is calculated based on the Euclidean distance between the actual outcome and the predicted probability assigned to the outcome for each observation, with low values being desirable. In addition, decision curve analysis was used to demonstrate the decision benefit based on the models.Several commonly used indices, including the AUC, sensitivity, specificity, and F1 score, were introduced to quantitively evaluate the predictive values of the models. To further demonstrate the favorable prediction performances of the ML-based models, the ML models and traditional severity assessment systems SAPS-II and LODS were compared in this study.