Odinel died in 1182 and was succeeded by his son Richard. Richard became one of the barons who stood against King John, and as a result forfeited his estates to the crown. They remained forfeited until 1217, the year after King John's death. Richard died in 1226 and was succeeded by his son, Gilbert, who was himself succeeded in 1245 by his son Gilbert. Through his mother, Gilbert II inherited the title of Earl of Angus, with vast estates in Scotland, but he continued to spend some of his time at Prudhoe. It is believed that he carried out further improvements to the castle. Gilbert took part in the fighting between Henry III of England and his barons, and in the Scottish expeditions of Edward I. He died in 1308 and was succeeded by his son, Robert D’Umfraville IV. In 1314, Robert was taken prisoner by the Scots at Bannockburn, but was soon released, though he was deprived of the earldom of Angus and of his Scottish estates. In 1316 King Edward granted Robert 700 marks to maintain a garrison of 40 men-at-arms and 80 light horsemen at Prudhoe.