After around 1200 the castle declined in importance: it was poorly situated, the threat of Danish raids had now passed, and so the nearby manor of Burstwick became the new administrative centre for the lordship instead. In January 1220 William de Forz, the Count of Aumale by marriage, rebelled against Henry III; part of their dispute involved the ownership of the estate of Driffield, away from Skipsea Castle, which Henry had seized the previous year, but William had been in disagreement with Henry's policies for several years before.