Williams was much more successful in fishing. An avid and expert fly fisherman and deep-sea fisherman, he spent many summers after baseball fishing the Miramichi River, in Miramichi, New Brunswick. Williams was named to the International Game Fish Association Hall of Fame in 2000. Williams, Jim Brown, Cumberland Posey, and Cal Hubbard are the only athletes to be inducted into the Halls of Fame of more than one professional sport. Williams was also known as an accomplished hunter; he was fond of pigeon-shooting for sport in Fenway Park during his career, on one occasion drawing the ire of the Massachusetts Society for the Prevention of Cruelty to Animals.