It passed from the Oger family (1127-1280) to the Montléon family (1280-1519) and eventually to the Chasteigner family (1519-1821).
The Oger family held ownership from 1127 to 1280, followed by the Montléon family from 1280 to 1519.  Finally, the Chasteigner family took over in 1519 and continued until 1821.