The first documentary record of the castle dates from 1086 and further evidence is limited until the 13th century. Robert's son, William, rebelled against Henry I of England in 1106, who confiscated the castle. Reginald de Dunstanville held it between 1141 and 1175, and it passed to the then Prince John when he acquired the title of the Count of Mortain in 1189, and then passed back into the hands of the Crown after John's rebellion against his brother, Richard I, in 1191. King John gave the castle to Hubert de Burgh, sheriff of Cornwall.