Carter was born in Washington, DC, the second of his parents' five children. He was raised in a family committed to public service. His mother worked as an executive assistant for Julian Bond and M. Carl Holman of the National Urban Coalition. An attorney turned administrator, his father was Executive Director of the Washington Urban League, and later a vice president at Cornell University. Carter's grandfather was a successful dentist in Harlem and his grandmother, Eunice Hunton Carter, was the first black woman to be a district attorney in New York state.