Richmond was part of the Phelps and Gorham Purchase. The area was first settled around 1790. The town was established in 1796 as "Pittstown", after Captain Peter Pitts, but was later renamed "Honeoye." In 1815, the name was changed to "Richmond", after Peter Pitts' wife, Abigail Richmond Pitts, a direct descendant of the Pilgrims. More territory was added to Richmond from the Town of Canadice in 1836. Parts of the Towns of Bristol and South Bristol were added to Richmond in 1848, but were returned to the source towns in 1852.