A detailed survey of the castle was made at the time of the exchange. This describes some of the buildings as 'in decay'. The castle itself was probably never inhabited again. It passed through various hands to William Albany, a London merchant taylor, but he and his descendants (from 1750 the Lloyd family of Aston near Oswestry, who still own the castle) lived at Great Fernhill. William's grandson, Francis Albany, fell into debt and sold his wood in Babbinswood to Arthur Kynaston of Shrewsbury, who built a forge at Fernhill, using stone from the castle. In 1632, the Castle Gatehouse was let, the tenant being allowed to take 'freestone out of the castle'. By the time of the English civil war, Whittington Castle was evidently no longer defensible and there is no evidence that it played any role in that war. In 1673, the castle (or rather the gatehouse) was let as a romantic dwelling to one Thomas Lloyd, a London merchant, probably retired. About 1760, one of the towers fell into the moat. This and other parts of the castle were used to make roads, probably including the new turnpike road to Ellesmere in 1776, during the minority of William Lloyd.