BackgroundRGB photographs are a powerful tool for dynamically estimating crop growth. Leaves are related to crop photosynthesis, transpiration, and nutrient uptake. Traditional blade parameter measurements were labor-intensive and time-consuming. Therefore, based on the phenotypic features extracted from RGB images, it is essential to choose the best model for soybean leaf parameter estimation. This research was carried out to speed up the breeding procedure and provide a novel technique for precisely estimating soybean leaf parameters.ResultsThe findings demonstrate that using an Unet neural network, the IOU, PA, and Recall values for soybean image segmentation can achieve 0.98, 0.99, and 0.98, respectively. Overall, the average testing prediction accuracy (ATPA) of the three regression models is Random forest > Cat Boost > Simple nonlinear regression. The Random forest ATPAs for leaf number (LN), leaf fresh weight (LFW), and leaf area index (LAI) reached 73.45%, 74.96%, and 85.09%, respectively, which were 6.93%, 3.98%, and 8.01%, respectively, higher than those of the optimal Cat Boost model and 18.78%, 19.08%, and 10.88%, respectively, higher than those of the optimal SNR model.ConclusionThe results show that the Unet neural network can separate soybeans accurately from an RGB image. The Random forest model has a strong ability for generalization and high accuracy for the estimation of leaf parameters. Combining cutting-edge machine learning methods with digital images improves the estimation of soybean leaf characteristics.Supplementary InformationThe online version contains supplementary material available at 10.1186/s13007-023-01023-z.