James Agar, 1st Viscount Clifden was the son of Henry Agar of Gowran Castle and the elder brother of Charles Agar, 1st Earl of Normanton. His mother was Anne Ellis, daughter of Welbore Ellis, Bishop of Meath, and sister of the politician Welbore Ellis, who in 1794 was created "Baron Mendip" in the Peerage of Great Britain with remainder to his three nephews Lord Clifden, the future Lord Normanton and another brother of theirs. Lord Clifden was succeeded by his son, the second Viscount. He represented County Kilkenny in the Irish House of Commons and Heytesbury in the British House of Commons.