Work continued under Robert's son, John Manners, who inherited the property as a minor in 1354. In 1355, when Sir Edward de Letham acquired the wardship of the property, the site was described as a "fortalice", a weakly defended location, but by 1368, when the wardship passed to Joan, his widow, it was considered to be a fully-fledged castle. De Letham was a powerful regional figure, and may have been granted the wardship by Edward III in order to retain his loyalty in the face of military advances by the Scots; he and his wife allegedly ran down the surrounding estate, resulting in a commission being established to investigate the abuses.