After the defeat of Llewelyn ap Gruffydd in 1282, the castle became a lordly residence for the FitzWarin family. However, after the death of Fulk VII in 1349, the castle went through a long period when the lords were almost always under age and usually absentees, though some repairs were carried out in about 1402. The lordship was laid waste in 1404 during the rebellion of Owain Glyndŵr, so that the lordship was worth nothing in 1407. However, the castle was not captured.