By the early 12th century the castle was controlled by the royal castellan, Simon de Beauchamp, the son of Hugh de Beauchamp who had helped conquer England in 1066. Contemporaries described the castle around this time as "completely ramparted around with an immense earthen bank and ditch, girt about with a wall strong and high, strengthened with a strong and unshakeable keep". Simon died in 1137, and King Stephen agreed that Simon's daughter should marry Hugh the Pauper and that the castle would be given to Hugh, in exchange for Stephen giving Miles compensatory honours and gifts. Miles and Payn de Beauchamp, the children of Simon's brother, Robert de Beauchamp, declared that the castle was rightfully Miles' and refused to hand it over to Hugh.