1557 saw the death of the 4th Lord Hay. His son, also John, abandoned the castle and moved into a new tower house on the site of the present day mansion of Yester House. In 1646 the 8th Lord was created Marquess of Tweeddale. The castle gradually fell into disrepair, and by the late 17th century was in a very parlous state, the stones having been much quarried for building material. Although the castle almost disappeared completely, Sir Hugo's original Goblin Ha' was tenanted by the Marquess' falconer until 1737. Yester House with its Adam interiors was sold in 1972 to Italian/American operatic composer Gian-Carlo Menotti then left to his son, Chip, who sold it in 2013.