The castle is recorded as having fallen into disrepair in a 1337 survey of the possessions of the Duchy of Cornwall. It was extensively repaired by order of the Black Prince but declined again following his death in 1376. By the time the antiquary John Leland saw it in the 16th century, it had fallen into ruin and had been extensively robbed for its stonework; as he put it, "the timber rooted up, the conduit pipes taken away, the roofe made sale of, the planchings rotten, the wals fallen down, and the hewed stones of the windowes, dournes, and clavels, pluct out to serve private buildings; onely there remayneth an utter defacement, to complayne upon this unregarded distresse."