Soil erosion (SE) is one of the most serious disasters in the world, which directly damage the productivity of the land and affect human well-being. How to effectively mitigate soil erosion is a challenge faced by all countries in the world. In this study, soil erosion was quantitatively assessed base on the RULSE model in an ecologically fragile area [Xiushui watershed (XSW)], and the effects of three major categories of factors (land use/cover change, landscape fragmentation and climate) on soil erosion were investigated using correlation analysis and structural equation model. The results indicated that there was no continuous increase or decrease trend on the SE of XSW with impact of rainfall, the mean values of SE were 2205.27 t/ha, 3414.25 t/ha and 3319.44 t/ha from 2000 to 2020 and the hot areas of SE were mainly distributed around the Xiushui river channel, respectively. The expansion of urbanization (the area of impervious increased from 113.12 to 252.57 km2) aggravated landscape fragmentation, and the landscape fragmented area had some overlap with the hot zone of SE. Additionally, the LUCC factor dominated by NDVI, landscape fragmentation factor and climate factor dominated by rainfall had a directly driving effect on SE, where the path coefficient of landscape fragmentation was 0.61 (P< 0.01), respectively. We also found that except increasing forest area, improving forest quality (NDVI, canopy closure, structure) deserved emphasized in SE management, and the effect of landscape fragmentation on SE also should not be ignored. Moreover, soil erosion assessment at large scales over long time periods tends to underestimate the driving force of rainfall on SE, and it is a great challenge to evaluate the effect of extreme rainfall on soil erosion at short time scales in a downscale manner. This research provides insights for ecological sustainable management and soil erosion protection policies.