Down was an advocate for higher education for women and disagreed with the notion that it would make the women liable to produce “feeble-minded” children. He also believed women should be allowed to join the workforce. This belief led him to petition the lords of Earlswood to pay his wife Mary for her contributions to the running of Earlswood. This request was refused because at the time the contributions of women in the workforce were considered volunteer work. This situation was thought to put a strain on Down's relationship with the lords of Earlswood. Down eventually retired from Earlswood in 1868 after the lords refused to give him the money he needed to display the artwork of some of his patients at an exhibition.