The last direct descendant of the Durbans to own the castle was Joseph de Gléon, Baron of Durban, who died aged 82 without heirs in 1787. The last inheritor of the castle sold it in 1873 to Paul Combes, who intended to use the stone as building material and had destroyed large parts of the structure by the end of the 19th century.
It was abandoned and then occupied by a Spanish soldier, who lived in the one room remaining. The castle subsequently became public property and continued to be used as a source of stone.