Skipsea Mere was drained in the second half of the 14th century and by 1397 the castle was considered worthless: the of land around it became used for pasturing animals. The counts of Aumale used the manor house at nearby Cleeton when they visited the area. Probably because of its poor location, the town of Skipsea Brough also proved unsuccessful as a commercial site. There were only three burgesses in the town paying rent in 1260, and by the late 14th century the town was largely abandoned; in 1377 there were only 95 people registered for the poll tax in the two settlements of Skipsea and Skipsea Brough combined.