The first castle on the site, built by Henry de Hilton in about 1072, was likely to have been built of wood. It was subsequently re-built in stone by Sir William Hylton (1376–1435) as a four-storey, gatehouse-style, fortified manor house, similar in design to Lumley and Raby. Although called a gatehouse, it belongs to a type of small, late-14th-century castle, similar to Old Wardour, Bywell and Nunney castles. The castle was first mentioned in a household inventory taken in 1448, as "a gatehouse constructed of stone" and although no construction details survive, it is believed the stone castle was built sometime between 1390 and the early 15th century, due to the coat of arms featured above the west entrance (see Heraldry below). It has been suggested that Sir William intended to erect a larger castle in addition to the gatehouse, but abandoned his plan.