During summer vacation Doby played baseball with a black semi-pro team, the Smart Sets, where he played with future Hall of Fame shortstop Monte Irvin. He also had a brief stint with the Harlem Renaissance, a professional basketball team, as an unpaid substitute player. Upon completing high school, he accepted an athletic scholarship to play basketball at Long Island University Brooklyn (LIU). Doby had been dating Eastside classmate Helyn Curvy since his sophomore year and, according to Doby, being able to remain close to Paterson was the "main reason" he selected LIU.