In 1165 Henry II conferred the castle on Roger de Powys, to whom he gave funds for its repair in about 1173. Roger was followed by his son Meurig (or Maurice), he was followed by his son Werennoc. A rival claim from Fulk III FitzWarin (who apparently claimed it under the Peverels) was not recognised until 1204, leading him to rebel against King John. FitzWarin was pardoned, and the castle and lordship of Whittington, not including Overton Castle was given to him. The castle then descended in the FitzWarin family, all called Fulk, until the death of Fulk XI in 1420.