Prior to the building of the hall, the land around Whiteley Wood belonged to the Mitchell family. By the time of the reign of Charles I ownership had switched to Thomas Dale and his family. Dale was a substantial freeholder in the Manor of Ecclesall bierlow and he owned a house on the site. Dale had two daughters who were his co-heirs, Anne married John Bright of Banner Cross Hall while Alice married Alexander Ashton of Stoney Middleton on 18 May 1659. It was Alexander Ashton who built Whiteley Wood Hall, it being completed in 1663, by which time he and Alice had two daughters. The Ashton family lineage remained at the hall until 1741 with Sarah Ashton being the last of the family to live there, having married Robert Alsibrook in 1709.