In the 1860s, Colyer took a yachting trip up the Connecticut shore as far as New Haven, looking for a good spot to relocate his home and studio. He liked what he saw at one island and bought there. Colyer later renamed the isle "Contentment Island", still its name. (According to one town history, the former name, stated in old land records, was "Ox Pound", another gives it as "Contention Island.") The artist took an active part in civic affairs and served a term in the state House of Representatives.