The castle (along with other Boyle properties – Chiswick House, Burlington House, Bolton Abbey and Londesborough Hall) was acquired by the Cavendish family in 1753 when Lady Charlotte Boyle (1731-1754), the daughter and heiress of The 3rd Earl of Burlington and 4th Earl of Cork, married the Marquess of Hartington, who later became, in 1755, The 4th Duke of Devonshire (1720-1764), a future Prime Minister of the Kingdom of Great Britain. Their son, the 5th Duke (1748–1811), carried out improvements at Lismore, notably the bridge across the River Blackwater in 1775 which was designed by Cork-born architect Thomas Ivory.