The castle saw its last military event in 1746, when a unit of government troops, of the Argyll and Sutherland Highlanders, occupied the building, causing further damage. By 1764, the York Buildings Company was bankrupt, and the remaining contents of the castle, including the roofs, were removed and sold to pay the company's creditors. The avenue of beech trees, which linked the castle and the village, was felled, and the property was sold to William Maule, Earl Panmure, nephew of the attainted Earl of Panmure. He died in 1782, and the property passed to his nephew, George Ramsay, 8th Earl of Dalhousie.