On October 26, 1847, Noyes was arrested for adultery, but was "released until his trial before the County Court in April next," according to Harriet H. Skinner's October 29, 1847 letter to her mother. Upon receiving word that arrest warrants had been issued for several of his loyal followers, the group left Vermont for Oneida, New York, where Noyes knew some friendly Perfectionists with land. They settled there, and built their first communal dwelling in 1848. In 1862, they built a larger communal home, which they called the Mansion House.