Fisk was born in Bellows Falls, Vermont, but according to Fisk, that was only because Vermont had the nearest hospital to his hometown, Charlestown, New Hampshire. He was raised by his parents, Leona and Cecil, along with older brother Calvin, younger brothers Conrad and Cedric and younger sisters Janet and Julie. He grew up in Charlestown, across the Connecticut River from Bellows Falls. Fisk attended Charlestown High School, where he played baseball (under Coach Ralph Silva), soccer, and basketball. Because his family is from New Hampshire, he insisted that the organization remove from his plaque in the Red Sox Hall of Fame its characterization of him as a Vermont native. Fisk earned his longtime nickname, "Pudge", because he was a chubby youngster.