After the death of the 6th Duke of Devonshire — known as the "Bachelor Duke" — in 1858, Lismore remained substantially unaltered. It became the home of a younger son of the 9th Duke, Lord Charles Cavendish, who married Adele Astaire, the sister and former dancing partner of Fred Astaire. After her husband's death in 1944 and her remarriage in 1947, Adele continued to use the castle until shortly before her own death in 1981. The castle was only used by The 11th Duke of Devonshire for brief annual visits, generally over Easter. In the last years of his tenure it was made available for short-term rent.