The castle was part of the manor of the Scrope family until 1520, when it passed to Elizabeth, the daughter of Thomas Scrope, wife of Sir Ralph Fitz Randolph.
Before 1520, the castle belonged to the Scrope family as part of their estate. It then changed hands to Elizabeth, the daughter of Thomas Scrope and wife of Sir Ralph Fitz Randolph.