King James I stayed in the castle in 1617. In the mid-17th century, Lady Anne Clifford made the castle her home. The castle was partly dismantled following a siege by roundhead forces in 1648, during the Second English Civil War. However, it was restored by Lady Anne Clifford in 1651–53. On her death the castle passed to the Earls of Thanet. They were responsible for converting the hall block into a classical mansion house. The upper parts of Caesar's Tower were altered in the 17th and 18th centuries. The house was largely rebuilt in 1686 and the northwest wing was added in 1695. In the 19th century it was again restored and sash windows were inserted.