Legume crop rotation is often adopted in rice cultivation to improve soil productivity. However, little is known about the role of microbes under legume rotation in affecting soil productivity. To elucidate this, a long-term paddy cropping experiment was set up to study the relationship between crop yield, soil chemical properties, and key microbial taxa under a double-rice and milk vetch rotation. Milk vetch rotation significantly improved soil chemical properties compared to no fertilization treatment, and soil phosphorus was a major factor correlated with crop yield. Long-term legume rotation increased soil bacterial alpha diversity and changed soil bacterial community. After milk vetch rotation, the relative abundances ofBacteroidota,Desulfobacterota,Firmicutes, andProteobacteriaincreased while those ofAcidobacteriota,Chloroflexi, andPlanctomycetotadecreased. Moreover, milk vetch rotation increased the relative abundance of phosphorus-related gene K01083 (bpp), which was significantly correlated with soil phosphorus content and crop yield. Network analysis showed that taxa ofVicinamibacteraleswere positively correlated with total phosphorus and available phosphorus, which was a potential taxon contributing to the availability of soil phosphorus stock. Our results indicated that milk vetch rotation could enrich key taxa with latent phosphate-solubilizing ability, increase the content of soil available phosphorus, and finally enhance crop yield. This could provide scientific guidance for better crop production.