The castle was built in 1070 by Hugh d'Avranches, the second Earl of Chester. It is possible that it was built on the site of an earlier Saxon fortification but this has not been confirmed. The original structure would have been a motte-and-bailey castle with a wooden tower. In the 12th century the wooden tower was replaced by a square stone tower, the Flag Tower. During the same century the stone gateway to the inner bailey was built. This is now known as the Agricola Tower and on its first floor is the chapel of St Mary de Castro. The chapel contains items of Norman architecture. In the 13th century, during the reign of Henry III, the walls of an outer bailey were built, the gateway in the Agricola Tower was blocked up and residential accommodation, including a Great Hall, was built along the south wall of the inner bailey. Later in the century, during the reign of Edward I, a new gateway to the outer bailey was built. This was flanked by two half-drum towers and had a drawbridge over a moat deep. Further additions to the castle at this time included individual chambers for the King and Queen, a new chapel and stables.