The castle stands on a rocky outcrop that dominates the town. The castle rock is geologically part of Mount Saint Sauveur, to the North West of the town and was separated from it due to erosion by the river Arget. Most of the present building dates from the 12th to 14th centuries. The Arget tower to the North was covered by a slate roof at the end of the nineteenth century, and is certainly the oldest dating to the 11th and 12th centuries.