The modifier of road materials from agricultural waste (AW) as raw material has been widely noticed. Considering the environmental impact of AW treatment and the National policy on the promotion of resource reuse, the feasibility of four AW (namely, bamboo powder, rape straw, corn cob, and wheat straw) for styrene butadiene styrene (SBS) asphalt modification is studied from the properties and mechanism perspectives. Through properties evaluation tests (such as the dynamic shear rheometer, multiple stress creep recovery, and rotating thin film oven test), the influence of four AW and different mixing amounts on the properties of SBS modified asphalt pavement is analyzed from the aspects of high-temperature deformation resistance and anti-aging properties. The results reveal that the four AW can improve the SBS asphalt’s high-temperature deformation resistance and anti-aging properties, among which rape straw has the most significant improvement effect. In addition, through the fourier transform infrared spectroscopy test, the microscopic mechanism of the AW/SBS composite modified asphalt binder is revealed from the functional groups. The analysis shows that the AW is physically mixed with the SBS asphalt binder, which inhibits the growth of sulfoxide groups and the cracking of the SBS modifier during aging.