The lordship of the manor, together with the initial manor house in Woolley, is first documented as owned by the Popeley family, of whom Robert Popeley was the last to reside. On his death he passed the building down to his only daughter Christine, who married into the Rilston family. The manor house remained with this line through four generations to her great grandson Robert Rilston, the son of Edmund. In around 1490, Robert sold this house, together with the lordship of the Manor of Woolley, to Sir Richard Woodrove (a descendant of John Woodrove), who added them to his existing estate.