Robert de Cardinham, lord of the manor between 1192–1225, then built up the inner curtain walls and converted the gatehouse completely to stone, giving the castle its current design. The village of Lostwithiel was established close to the castle at around the same time. The castle belonged to the Cardinhams for several years, who used it in preference to their older castle at Old Cardinham. Andrew de Cardinham's daughter, Isolda de Cardinham, finally married Thomas de Tracey, who then owned the castle until 1264.