Henry VIII gave the ruins to the Carey family, who retained it until it passed by marriage into first the Heviningham and then the Coke families. The castle was not involved in the events of the English Civil War in the 17th century, and escaped the slighting that affected many similar properties, probably because the collapse of the outer walls had already made it indefensible and of little military value. In 1737, after the death of Edward Coke, the castle and the surrounding manor were bought by Thomas Osborne, the Duke of Leeds, for £22,500.