Self-compacting mortar (SCM) has superior workability and long-term durable performance compared to traditional mortar. The strength of SCM, including both its compressive and flexural strengths, is a crucial property that is determined by appropriate curing conditions and mix design parameters. In the context of materials science, predicting the strength of SCM is challenging because of multiple influencing factors. This study employed machine learning techniques to establish SCM strength prediction models. Based on ten different input parameters, the strength of SCM specimens were predicted using two different types of hybrid machine learning (HML) models, namely Extreme Gradient Boosting (XGBoost) and the Random Forest (RF) algorithm. HML models were trained and tested by experimental data from 320 test specimens. In addition, the Bayesian optimization method was utilized to fine tune the hyperparameters of the employed algorithms, and cross-validation was employed to partition the database into multiple folds for a more thorough exploration of the hyperparameter space while providing a more accurate assessment of the model’s predictive power. The results show that both HML models can successfully predict the SCM strength values with high accuracy, and the Bo-XGB model demonstrated higher accuracy (R2= 0.96 for training and R2= 0.91 for testing phases) for predicting flexural strength with low error. In terms of compressive strength prediction, the employed BO-RF model performed very well, with R2= 0.96 for train and R2= 0.88 testing stages with minor errors. Moreover, the SHAP algorithm, permutation importance and leave-one-out importance score were used for sensitivity analysis to explain the prediction process and interpret the governing input variable parameters of the proposed HML models. Finally, the outcomes of this study might be applied to guide the future mix design of SCM specimens.