The house was remodelled and partially rebuilt in 1832-3 by Sir Robert Smirke for the industrialist James Foster. His nephew William Orme Foster lived there until 1868, when he sold the lease. Around 1890, Martha Steer (née Nettlefold) took it on lease for herself, her daughter, Ethel Steer, and the Wymans with their family of young children. The castle was vacant from the death of George Arkle until the sale of the Prestwood estate in 1913. It was bought by Francis Grazebrook, a relative of the earlier tenant, and remained in the family until the death of his son O. F. Grazebrooke in 1974. The 19th century main front incorporates a late medieval gate tower.