He married twice; firstly on 3 April 1745 to Marie Charlotte de La Tour d'Auvergne (20 December 1729 – 6 September 1763), daughter of Emmanuel Théodose de La Tour d'Auvergne and his last wife Louise Henriette Françoise of Lorraine. The couple had one child who married into the Noailles family. His first wife Marie Charlotte died of Smallpox aged 33.