Microbial diversity is an important indicator of soil fertility and plays an indispensable role in farmland ecosystem sustainability. The short-term effects of fertilization and rhizobium inoculation on soil microbial diversity and community structure have been explored extensively; however, few studies have evaluated their long-term effects. Here, we applied quantitative polymerase chain reaction (qPCR) and amplicon sequencing to characterize the effect of 10-year fertilizer and rhizobium inoculation on bacterial communities in soybean bulk and rhizosphere soils at the flowering–podding and maturity stages. Four treatments were examined: non-fertilization control (CK), phosphorus and potassium fertilization (PK), nitrogen and PK fertilization (PK + N), and PK fertilization andBradyrhizobium japonicum5821 (PK + R). Long-term co-application of rhizobium and PK promoted soybean nodule dry weight by 33.94% compared with PK + N, and increased soybean yield by average of 32.25%, 5.90%, and 5.00% compared with CK, PK, and PK + N, respectively. The pH of PK + R was significantly higher than that of PK and PK + N at the flowering–podding stage. The bacterial abundance at the flowering–podding stage was positively correlated with soybean yield, but not at the maturity stage. The significant different class Gemmatimonadetes, and the generaGemmatimonas, andEllin6067in soil at the flowering–podding stage were negatively correlated with soybean yield. However, the bacterial community at class and genus levels at maturity had no significant effect on soybean yield. The key bacterial communities that determine soybean yield were concentrated in the flowering–podding stage, not at maturity stage. Rhizosphere effect, growth period, and treatment synergies resulted in significant differences in soil bacterial community composition. Soil organic matter (OM), total nitrogen (TN), pH, and available phosphorus (AP) were the main variables affecting bacterial community structure. Overall, long-term co-application of rhizobium and fertilizer not only increased soybean yield, but also altered soil bacterial community structure through niche reconstruction and microbial interaction. Rhizobium inoculation plays key role in reducing nitrogen fertilizer application and promoting sustainable agriculture practices.