The first known settlement occurred in 1804. The island was once considered to be part of Canada but, since 1823, has been recognized as part of the United States. Shortly afterwards a dispute over some harvested trees created a brief disturbance called "The Grindstone Island War" in which American militia attempted to prevent the removal of the logs. In 1831 the entire island was purchased by Eliza Evertson, the widow of Nicholas Evertson and the grandmother of Edgar Saltus, for $15,000. The Evertson family owned the island for over 40 years.