After his second marriage in 1861, Wheaton and his large family first took over the second floor of the American House Hotel, built by John North in 1857. (The hotel became the first building of Northfield College, later renamed Carleton College.) The family built a new house of Greek Revival style at 405 Washington Street, where they moved in 1868. The house was divided and moved in 1938. Both portions of the original house still stand in Northfield. The main house is about five blocks south of its original location and the "L" portion is on 9th Street West.