Doby was born in Camden, South Carolina, to David Doby and Etta Brooks. Doby's father served in World War I. He worked as a horse groomer and played semi-pro baseball, but drowned in an accident at age 37 in New York state. Doby's mother, who had divorced David before his death, moved to Paterson, New Jersey; Doby remained in Camden. He lived with his grandmother before moving to live with his father's sister and brother-in-law from 1934 to 1938. He attended Jackson School, which was segregated under South Carolina state law. His first opportunity to play organized baseball came as a student at Boylan-Haven-Mather Academy, a private school affiliated with the Methodist church. Richard Dubose, who had managed Doby's father and was known locally in African-American circles for his baseball expertise, gave Doby some of his first baseball lessons. Reflecting on his years growing up in South Carolina, including how he and playmates used worn down broom handles for bats, Doby said, "Growing up in Camden, we didn't have baseball bats. We'd use a tree here, a tin can there, for bases."