The original stone keep, built before 1267, is generally considered to be by Sir Hugo de Giffard. A grandson of the first Laird of Yester, he served as a guardian of the young Alexander III of Scotland, and was by repute a magician and necromancer. Alexander III is known to have been at Yester on and around May 24, 1278, where he corresponded with Edward I of England. Following the Scots Wars of Independence, Yester was rebuilt as a castle of "enceinte".