Following the restoration of Episcopacy to the Scottish Church in 1662 ownership of the castle passed back to the church, but it was starting to fall into decay. Parliament granted Bishop Murdo MacKenzie £1000 for repairs and this sustained the building up to 1689 when the last occupant, Bishop William Hay was expelled after refusing to take an oath of allegiance to King William and Queen Mary. The palace passed into the hands of the Crown and the fine ironwork and wood carvings removed. Local people plundered the walls for stonework for building works until the early 19th century when it passed in the ownership of James Dunbar-Brander of Pitgaveny. It was leased to the Departement of the Works in 1974. The tower by that time had a pronounced bulge in the East wall - the structure being undermined by attempts by covenanters to blow it up.