During the 17th century, the priory became ruinous and was used as a quarry for the rebuilding of St Giles Kirk, in Elgin. The payments made to hauliers imply that the pulling down of the building must have been on a large scale. The lands at Pluscarden came into the ownership of the Earls of Fife and the 4th Earl arrested further deterioration when he converted the east range into a shooting lodge. The property was then bought by John Patrick Crichton-Stuart, 3rd Marquess of Bute in 1897 who commenced restoration of the church but was halted in 1900 upon his death.