PurposeYoung age at breast cancer (BC) diagnosis and family history of BC are strongly associated with high prevalence of pathogenic variants (PVs) inBRCA1andBRCA2genes. There is limited evidence for such associations with moderate/high penetrance BC-risk genes such asATM,CHEK2, andPALB2.MethodsWe analyzed multi-gene panel testing results (09/2013–12/2019) for women unaffected by any cancer (N = 371,594) and those affected with BC (N = 130,151) ascertained for suspicion of hereditary breast and/or ovarian cancer. Multivariable logistic regression was used to test association between PV status and age at BC diagnosis (≤ 45 vs. > 45 years) or family history of BC after controlling for personal/family non-BC histories and self-reported ancestry.ResultsAn association between young age (≤ 45 years) at diagnosis and presence of PVs was strong forBRCA1(OR 3.95, 95% CI 3.64–4.29) and moderate forBRCA2(OR 1.98, 95% CI 1.84–2.14). Modest associations were observed between PVs and young age at diagnosis forATM(OR 1.22, 95% CI 1.08–1.37) andCHEK2(OR 1.34, 95% CI 1.21–1.47) genes, but not forPALB2(OR 1.12, 95% CI 0.98–1.27). For women with BC, earliest age of familial BC diagnosis followed a similar pattern. For unaffected women, earliest age of family cancer diagnosis was significantly associated with PV status only forBRCA1(OR 2.34, 95% CI 2.13–2.56) andBRCA2(OR 1.25, 95% CI 1.16–1.35).ConclusionsYoung age at BC diagnosis is not a strong risk factor for carrying PVs in BC-associated genesATM, CHEK2,orPALB2.Supplementary InformationThe online version contains supplementary material available at 10.1007/s10549-023-06946-8.