After completing eighth grade, Doby moved north to Paterson at the age of 14 to be reunited with his mother; she visited him weekly while he lived with one of her friends. At Paterson Eastside High School, Doby was a multi-sport athlete; as well as playing baseball and basketball, he was a wide receiver in football and lettered in track. After winning a state football championship, the Eastside team was invited to play in Florida, but the promoters would not allow Doby, the only black player on the team, to participate. Consequently, the team voted to forgo the trip as a gesture of support for Doby.