Fisk excelled at basketball. When he was a sophomore, he was a starter and helped Charlestown to an undefeated season and the 1963 New Hampshire Class M championship. His play in a 1965 high school basketball tournament in the Boston Garden drew the attention of Boston Celtics owner Walter Brown, who told a local reporter, "You have got to tell me—who is that kid?" Fisk had 42 points and 38 rebounds in the 1965 state tournament semifinal loss to Hopkinton High School. He was awarded a basketball scholarship by the University of New Hampshire, where he started for the UNH Wildcats while also playing baseball. He met his wife Linda Foust while at UNH. The freshman team that Fisk played for was undefeated for the 1965–66 season. In 1966, he played collegiate summer baseball in the Cape Cod Baseball League for the Orleans Cardinals. In his sophomore year, the Red Sox drafted him in the first round of the January 1967 amateur draft, and his athletic future was set. Fisk gave up his dreams of basketball glory. "I could never be a six-foot-two power forward and play for the Celtics," he said.