Carey is a direct descendant of James K. Polk. Carey's family has long ties in Baltimore. His great-great-great-grandfather James Carey was an 18th- and 19th-century Baltimore shipper, chairman of the Bank of Maryland, a member of Baltimore's first City Council and a relative of Johns Hopkins. His grandmother, Anne Galbraith Carey, started the Gilman School for boys in Roland Park in 1897. 