In 1438, another of John Manners' sons, Robert, inherited the castle and the estates. The lands had fallen in value considerably over the preceding years, as a result of the wars with the Scots, the feuding with the Herons and the poverty of the lands. Robert's son, another Robert Manners, inherited the property in 1464, and passed the castle onto his son, George, on his death in 1495. George inherited the title of Baron de Ros from his mother's side of the family and the castle was abandoned as a residence by the family around the end of the 15th century when the Manners moved closer to the royal court.