The manor of Hedingham was awarded to Aubrey de Vere I by William the Conqueror by 1086. The castle was constructed by the de Veres in the late 11th and early 12th centuries, and the keep in the 1130s and 1140s. To accommodate the existing castle, a large ditch was cut through a natural spur westward into the Colne Valley in order to form a ringwork and inner bailey; an outer bailey extended south further into the valley and what is now the modern village of Castle Hedingham. The stone keep is the only mediaeval structure to survive, and is in an excellent state of preservation.