The castle began to decline in importance, however, and in 1369 passed to the Despensers and then the Beauchamps, neither of whom used the castle. It was temporarily refortified by Henry IV in response to the Owain Glyndŵr uprising in North Wales in 1403.
The castle's importance declined, with ownership passing from the original owners to the Despensers and then the Beauchamps, neither of whom used it.  Henry IV briefly refortified the castle in response to Owain Glyndŵr's uprising in North Wales in 1403.