A native of Oriskany Falls, New York, at the age of 20, Spooner was signed to a contract by the Brooklyn Dodgers as an amateur free agent in 1951 and was assigned to play in Minor league baseball. Spooner's early minor league career was promising, but erratic. Walks were frequently a problem, as Spooner averaged a walk per inning in his first two minor league seasons. However, he also had great speed, and showed flashes of brilliance. He made his way up the Dodgers' minor league system, compiling a fairly unimpressive 27–36 record during his first three seasons. He had a breakout year in 1954 when he won 21 games with 262 strikeouts in 238 innings for the Fort Worth Cats of the Texas League. His successful performance earned him a promotion to the Brooklyn Dodgers with one week left in the 1954 season.