Williams lived in San Diego's North Park neighborhood (4121 Utah Street). At the age of 8, he was taught how to throw a baseball by his uncle, Saul Venzor. Saul was one of his mother's four brothers, as well as a former semi-professional baseball player who had pitched against Babe Ruth, Lou Gehrig, and Joe Gordon in an exhibition game. As a child, Williams's heroes were Pepper Martin of the St. Louis Cardinals and Bill Terry of the New York Giants. Williams graduated from Herbert Hoover High School in San Diego, where he played baseball as a pitcher and was the star of the team. During this time, he also played American Legion Baseball, later being named the 1960 American Legion Baseball Graduate of the Year.