He was now premier baronet of England, and in the same year he was made Lord-Lieutenant of Buckinghamshire, being succeeded in the colonelcy of the militia by John Wilkes. As Lord Le Despencer he now sank into comparative respectability and insignificance. He took a disgraceful part with John Montagu, 4th Earl of Sandwich, in raking up charges against their common friend Wilkes in connection with the "Essay on Woman", and during Lord North's long administration from 1770 to 1781 he was joint Postmaster General. When, however, Chatham fell down in a swoon during his last speech in the House of Lords, Despencer was almost the only peer who came to his assistance. He died at West Wycombe after a long illness on 11 December 1781, and was buried in the mausoleum he had built there. His wife had died on 19 January 1769, and was also buried at Wycombe.