It was a Norman motte and bailey castle which was founded by Ralph de Buron. In the 13th century King John built a stone castle, adding a keep, a chapel, a gatehouse and a barbican.
In 1514 it was granted by Henry VIII to the Duke of Norfolk as a reward for services against the Scots. Later it came into the possession of the Stanhope family.