The first Cary castle was besieged in 1138 and again in 1152. It was originally made of timber frame, as it would have been built by the Normans and inherited their building traditions in wood. Cary first appears in recorded history after the Norman invasion in the 11th century though according to Castle Cary history 'there is evidence the site was occupied and fortified before this. The second castle was abandoned by the mid-15th century in favour of a manor house and the stones pillaged for other buildings in the town.