The predictive ability of coronary heart disease (CHD) and ischemic stroke (IS) polygenic risk scores (PRS) have been evaluated individually, but whether they predict the combined outcome of atherosclerotic cardiovascular disease (ASCVD) remains insufficiently researched. It is also unclear whether associations of the CHD and IS PRS with ASCVD are independent of subclinical atherosclerosis measures. 7,286 White and 2,016 Black participants from the population-based Atherosclerosis Risk in Communities study who were free of cardiovascular disease and type 2 diabetes at baseline were included. We computed previously validated CHD and IS PRS consisting of 1,745,179 and 3,225,583 genetic variants, respectively. Cox proportional hazards models were used to test the association between each PRS and ASCVD, adjusting for traditional risk factors, ankle-brachial index, carotid intima media thickness, and carotid plaque. The hazard ratios (HR) for the CHD and IS PRS were significant with HR of 1.50 (95% CI: 1.36–1.66) and 1.31 (95% CI: 1.18–1.45) respectively for the risk of incident ASCVD per standard deviation increase in CHD and IS PRS among White participants after adjusting for traditional risk factors. The HR for the CHD PRS was not significant with an HR of 0.95 (95% CI: 0.79–1.13) for the risk of incident ASCVD in Black participants. The HR for the IS PRS was significant with an HR of 1.26 (95%CI: 1.05–1.51) for the risk of incident ASCVD in Black participants. The association of the CHD and IS PRS with ASCVD was not attenuated in White participants after adjustment for ankle-brachial index, carotid intima media thickness, and carotid plaque. The CHD and IS PRS do not cross-predict well, and predict better the outcome for which they were created than the composite ASCVD outcome. Thus, the use of the composite outcome of ASCVD may not be ideal for genetic risk prediction.