In 1802, he succeeded his great-uncle Welbore Ellis, 1st Baron Mendip as second Baron Mendip according to the special remainder, and in 1804 assumed by royal license the additional arms and surname Ellis. The titles were to remain united until the extinction of the barony and viscountcy of Clifden in 1974. Lord Clifden's only son George James Welbore Agar-Ellis was created "Baron Dover" in 1831 (see below), but predeceased his father. He was therefore succeeded by his grandson, the third Viscount, who had already succeeded his father as second Baron Dover in 1833. His only son, the fourth Viscount, died unmarried at an early age, when the titles passed to his uncle, the fifth Viscount. He had previously represented County Kilkenny in Parliament as a Liberal.