After the suspicious death of his wife Drogo fled England, and the castle was reassigned by William the Conqueror to Odo, the Count of Aumale. In 1096 it passed to Arnulf de Montgomery, but returned to the Aumales in 1102, who held it until 1221. Trade initially flourished and as a result William le Gros founded the fortified town of Skipsea Brough along the ridgeway just south of the castle, probably around 1160. The town was intended to bring in valuable revenue to the earls, but would also have helped to defend the castle on its most vulnerable, overlooked side. The castle-guard system lapsed, with the surrounding estates paying their rents in cash instead.