Harold Godwinson repaired the town's defences in 1056, but what happened to the castle is not mentioned. Shortly after the Norman Conquest in 1066, William FitzOsbern was made Earl of Hereford. He may have rebuilt the castle as it was in use in 1067 when the "Anglo-Saxon Chronicle" noted that Eadric the Wild harassed its garrison. In 1071 William FitzOsbern died and his son Roger took over possession of the castle. He was involved in an unsuccessful attempt to depose King William and consequently forfeited the castle.