The accumulation of heavy metals in agricultural soils concerns food security. By using the Geographical Detector, this study investigated the influence of six types of factors (eleven factors) on the accumulation of Cd, Pb, Cu, Zn in agricultural soil and products of the North China Plain and confirmed the dominant factor. The results showed that heavy metals had accumulated in regional agricultural soils and the accumulation of Cd was severe. The accumulation of heavy metals was significantly influenced by policy factors (the management and reduction in usage of fertilizers and pesticides), fertilization factors (application of organic and chemical fertilizers), pesticide factors (application of herbicide and insecticide) and atmospheric deposition factors (heavy metal concentration in atmospheric deposition). The policy factor dominated the other three types of factors. Atmospheric deposition and the excess application of fertilizers and pesticides directly lead to the accumulation of heavy metals. Due to the high concentrations of heavy metals and abundant application amounts, organic fertilizers have contributed high levels of heavy metals to agricultural soils. This study suggests that formulated fertilization and action plans for pesticide reduction could effectively decrease the accumulation of heavy metals in agricultural soils and products in the study area.