It is believed that McDowell built the house around 1770; although no records exist to show this, deeds do show that it was that year he acquired from James Humphrey. McDowell was the son of James McDowell, a native of County Longford in Ireland who was one of the few survivors of a malnutrition-plagued Atlantic crossing in 1729 funded by Charles Clinton, grandfather of future New York State governor DeWitt. Bound for Pennsylvania, the survivors landed instead in Cape Cod. Two years later, in 1731, they made their way to the Hudson Valley where they were the first settlers in the area they called Little Britain.