Born in Los Angeles, Battey attended Jordan High School in the Watts neighborhood of Los Angeles. He was signed as an amateur free agent by the Chicago White Sox prior to the 1953 season. Battey was assigned to the Colorado Springs Sky Sox in the Single-A Western League where he began his baseball career by hitting only a .158 batting average in 26 games. The White Sox demoted him to the Waterloo White Hawks in the B-level Illinois–Indiana–Iowa League and Battey responded with a .292 average and 11 home runs in 129 games. In , Battey was promoted to the Triple-A Charleston Senators in the American Association where, he hit for a .269 batting average along with 8 home runs and 71 runs batted in. His performance earned him a promotion to the major leagues where he made his debut with the White Sox on September 10, 1955 at the age of 20.