When Bryan de Brampton died in 1294, his son-in-law, Robert Harley, inherited the castle. For almost 700 years since the castle has remained in the Harley family. During the Civil War, the castle was defended by Lady Brilliana Harley, the wife of Robert Harley. Harley was a puritan and a Member of Parliament.