The castle was originally a motte and bailey castle built in 1090 by Robert de Romille, lord of the multiple estates of Bolton Abbey. Shortly after 1102 Henry I extended Romille's lands to include all of upper Wharfedale and upper Airedale. The earth and wood castle was rebuilt in stone to withstand attacks by the Scots. The cliffs behind the castle, dropping down to Eller Beck, made the castle a perfect defensive structure. The Romille line died out, and in 1310 Edward II granted the castle to Robert Clifford who was appointed Lord Clifford of Skipton and Guardian of Craven. Robert Clifford ordered many improvements to the fortifications, but died in the Battle of Bannockburn in 1314 when the improvements were barely complete.