The 2nd Earl held political office as Lord-Lieutenant of Ireland and Lord Privy Seal and died unmarried. He was succeeded as 3rd Earl by his younger brother, who was a vice-admiral in the Royal Navy and also served as Chief Secretary for Ireland from 1766 to 1767. He also died without legitimate issue and was succeeded in December 1779 by the next younger brother, who thereby became the 4th Earl. Known as the "Earl-Bishop", the fourth Earl served as Church of Ireland Lord Bishop of Cloyne from 1767 to 1768 and as Lord Bishop of Derry from 1768 to 1803. Although an efficient clergyman, varying estimates have been found of his character. He had a reputation for high living, and hostelries in various countries took to calling themselves the ‘Hotel Bristol’, implying that they were the best in town. The custom continues today. In 1795, he began expanding his ancestral home, thus creating Ickworth House in its modern form. The house was still unfinished when he died in 1803 and was completed by his successor. In 1799, he became the fifth Baron Howard de Walden when the abeyance of this peerage was terminated. Bishop Lord Bristol married Elizabeth, sister and heir of Sir Charles Davers, 5th Baronet (c. 1730–1763), and great-granddaughter of Thomas Jermyn, 2nd Baron Jermyn (see the Baron Jermyn). His second son, John Augustus Hervey, Lord Hervey, was a captain in the Royal Navy and also served as ambassador to Florence. He predeceased his father. His daughter, the Hon. Elizabeth Catherine Caroline Hervey (1780–1803), married Charles Ellis, later first Baron Seaford. Their son Charles succeeded as sixth Baron Howard of Walden on the death of his great-grandfather, Lord Bristol, in 1803 (see Baron Howard de Walden and Baron Seaford). Upon the 4th Earl's death, the Bristol title passed to his third but eldest surviving son Frederick, who thereby became the 5th Earl. He was a politician and served under Henry Addington as Parliamentary Under-Secretary of State for Foreign Affairs from 1801 to 1803.