From 1812 to 1827, he served as Lord President of the Council under Lord Liverpool. After George Canning's death in 1827, Harrowby refused to serve George IV as prime minister and never held office again. Despite this he continued to take part in politics, being especially prominent during the deadlock which preceded the passing of the Reform Bill in 1832. Harrowby's long association with the Tories did not prevent him from assisting to remove the disabilities of Roman Catholics and Protestant dissenters, or from supporting the movement for electoral reform; he was also in favour of the emancipation of the slaves. He was a member of the Literary Association of the Friends of Poland.