It had been occupied during the minority of Fulk XI by his mother and her new husband William Lord Clinton, during whose time there was a dispute with the people of Oswestry who had cut down oak trees in his woods. When the FitzWarin line died out in 1420, the lordship passed to Fulk XI's sister Elizabeth, who married Richard Hankeford. In 1422, the castle was captured by escalade by William Fitzwaryn (presumably a cousin claiming the castle as heir male) and Richard Laken, but evidently soon restored to Lord Clinton. Their daughter Thomasia married William Bourghchier, thus carrying the FitzWarin peerage into the Bourchier family. Their grandson John Bourchier was made Earl of Bath, but his son John Bourchier, 2nd Earl of Bath exchanged the lordship and castle in 1545 with Henry VIII, for some former monastic estates nearer the main family home in Devon.