Particle media are widely used in engineering and greatly influence the performance of engineering materials. Asphalt mixtures are multi-phase composite materials, of which coarse aggregates account for more than 60%. These coarse aggregates form a stable structure to transfer and disperse traffic loads. Therefore, knowing how to adjust the structural composition of coarse aggregates to optimize their performance is the key to optimize the performance of asphalt mixtures. In this study, the effects of different roughness and different sizes on the interlocking force and contact force of coarse aggregates were investigated through means of simulation (DEM), and then the formation-evolution mechanism of the coarse aggregate structure and the role of different sizes of aggregates in the coarse aggregate structure were analyzed. Subsequently, the optimal ratio of coarse aggregates was explored through indoor tests, and finally, the gradation of asphalt mixture based on the optimization of fine structure was formed and verified through indoor tests. The results showed that the major model can effectively reveal the role of different types of aggregates in the fine structure and the relationship between the strength of contact forces between them and clarify that the strength of the fine structure increases with the increase in aggregate roughness. Hence, the coarse aggregate structure can be regarded as a contact force transmission system composed of some strong and sub-strong contact forces. Their formation-evolution mechanism can be regarded as a process of the formation of strong and sub-strong contact forces and the transformation from sub-strong contact force to strong contact force. Moreover, the dynamic stability of the optimized graded asphalt mixture was increased by 30%, and the fracture toughness was increased by 26%.