When the Whigs came to power under Lord John Russell in 1846, Lord Spencer was appointed Lord Chamberlain of the Household. He was sworn of the Privy Council the same year. He resigned as Lord Chamberlain in 1848 but returned to the government as Lord Steward of the Household in early 1854 under Lord Aberdeen, a post he held until shortly before his death in 1857, the last two years under the premiership of Lord Palmerston. He was made a Knight of the Garter in 1849.