The earliest part of the castle was its residential tower. This tower may have been built around 1341 on the site of an older, unfortified house owned by the family on the same site, incorporating part of the structure into the new, crenellated tower. Alternatively, the central tower may have been built at some point between the late 13th and early 14th centuries, complete with crenellations, in which case the licence from Edward III served only to allow Manners to extend the perimeter fortifications. By the 1350s, the castle was surrounded by a manor which included mills for corn and fulling, lime kilns and coal mines.