Non-native settlement in the region dates to ca. 1740. In 1754, prominent soldier and land owner Colonel Charles Broadwater settled within the town boundaries. Broadwater's son-in-law, John Hunter built the first recorded house there in 1767, naming it Ayr Hill to recall his birthplace, Ayr, Scotland. That name was then applied to the tiny developing community. The name of the town was changed in the 1850s, when a doctor, William Hendrick, settled there if the town renamed itself after his hometown, Phelps, New York, which was then known as Vienna.