After resigning from Earlswood, Down set up his own private home for those with developmental and intellectual disabilities at Normansfield, between Hampton Wick and Teddington. The home's first occupants were 18 mentally disabled children of upper-class members of the community such as lords and physicians. In the home, Down and his wife did their best to educate the children and exposed them to a wide variety of mentally stimulating activities. Normansfield was a success and eventually had to be expanded to house the growing number of its inhabitants. By 1876 the number of inhabitants in Normansfield had grown to 160.