After the Restoration of Charles II to the throne in 1660, a royal survey of the castle found the fortification to be in ruins. An increasing number of visitors came to see the castle in the 18th and 19th centuries, with the north-east corner becoming a popular site for picnics. Writing in 1785, the antiquarian Francis Grose attributed the decline of the fortification to the changes in the local harbours and the superiority of the British Navy in protecting the coasts, observing that the castle's architecture "clearly shew the low state of military architecture" during the 16th century in England.