In the late 14th century John de Crichton (d.1406) built a tower house here as his family residence. John's son, William (d. c. 1453), served as Lord Chancellor of Scotland, and was made Lord Crichton in c. 1443. In 1440 he had been partly responsible for organising the "Black Dinner", where the young William Douglas, 6th Earl of Douglas, was murdered. As a result, he obtained the Douglas property of Bothwell Castle in Lanarkshire for himself. John of Corstorphine, chief of Clan Forrester and a Douglas adherent, stormed and slighted the castle in 1445 in retaliation. William, however, reconstructed and extended the castle, and also built the nearby collegiate church. The 3rd Lord Crichton was a supporter of Alexander Stewart, Duke of Albany, and his lands and titles were forfeit in 1483, when Albany was sentenced for treason. Crichton Castle, along with Bothwell Castle, was briefly granted to Sir John Ramsey, who forfeited it in 1488.