Williams was born on May 7, 1929, in St. Louis, Missouri, and lived there until age 13, when his family moved to Pasadena, California. He attended Pasadena High School, and then enrolled in Pasadena City College. He signed his first professional contract with the Brooklyn Dodgers in 1947, and played his first major league game with Brooklyn in 1951. A right-handed batter and thrower, Williams was listed as tall and . Initially an outfielder, he separated a shoulder attempting to make a diving catch on August 25, 1952; he missed the rest of the season and permanently weakened his throwing arm. As a result, he learned to play several positions (he was frequently a first baseman and third baseman) and became a notorious "bench jockey" in order to keep his major league job. He appeared in 1,023 games over 13 seasons with the Dodgers, Baltimore Orioles, Cleveland Indians, Kansas City Athletics and Boston Red Sox. Williams posted a career batting average of .260; his 768 hits included 70 home runs, 157 doubles and 12 triples. In the field, he appeared in 456 games in the outfield, 257 at third base, and 188 at first.