At the beginning of the 15th century the castle was again attacked, this time by Owain Glyndŵr, who took control of the castle around 1403–05. Repairs to the castle continued until at least 1430, but just a century later the antiquary John Leland recorded that the castle was a ruin set in marshland, with a single tower being used as a prison. In the mid-16th century the 2nd Earl of Pembroke used the castle as a manorial court, but in 1583 the castle was leased to Thomas Lewis, who accelerated the castle's dilapidation by removing stonework to build his nearby manor, The Van. The Lewis family, who claimed descent from Ifor Bach, left the manor in the mid-18th century when they purchased St Fagans Castle, The Van falling into decay.