After Fitz-Ansculf, the castle came into the possession of the Paganel family, who built the first stone castle on the site. This castle was strong enough to withstand a siege in 1138 by the forces of King Stephen. However, after Gervase Paganel joined a failed rebellion against King Henry II in 1173 the castle was demolished by order of the king. The Somery's were the next dynasty to own the site when Ralph de Somery I succeeded his uncle, Gervase Paganel in 1194. Roger de Somery II set about rebuilding the castle in 1262. The castle was far from complete on the death of Roger de Somery II in 1272 and construction carried on from this time into the 14th century by Roger's heirs. The keep (the most obvious part of the castle when viewed from the town) and the main gate date from this re-building.