The current house was built by Gregory between 1832 and 1854, and it helped usher in a renaissance of Elizabethan architecture. The original architect, Anthony Salvin, was replaced by William Burn, who is responsible for its interior detailing. Upon Gregory's death, the manor passed to his cousin George Gregory and then in 1860 to a distant relative, John Sherwin-Gregory. Upon the death of Sherwin's wife in 1892, it passed to his godson Thomas Pearson-Gregory, who allowed it to fall into disrepair.