The castle, on a rock to the east of the town above All Saints' Church, was constructed in approximately 1070 by Ilbert de Lacy. on land which had been granted to him by William the Conqueror as a reward for his support during the Norman Conquest. There is, however, evidence of earlier occupation of the site. Initially the castle was a wooden structure which was replaced with stone over time. The Domesday Survey of 1086 recorded "Ilbert's Castle" which probably referred to Pontefract Castle.