A castle held by Geoffrey de Costentin is recorded as a rebel stronghold against Henry II in 1173–1174 when his sons revolted. There is an incorrect local tradition that Geoffrey was the king's son, Geoffrey II, Duke of Brittany, who was one of the rebels. Dent gives the size of the castle as about , and suggests it was similar in pattern to those at Pontefract and Launceston. The castle was probably ruinous by the middle of the 16th century, and in 1642 it was agreed to demolish it. Castle Hill, possibly the motte, was levelled in 1775 to make space for Warren's mill, see below. Nearby walls, once thought to be either part of the castle or of the town walls, are now thought to be revetments to protect the cliff face from erosion.