He held the offices of Clerk of the Household to the Prince of Wales (1756–60) and Clerk of the Green Cloth from 1761 to 1762. In 1765 he was also made a Lord of the Treasury in the first Rockingham ministry and continued in that office in the Pitt (then Lord Chatham) administration until December 1767, when he became a member of the Privy Council and joint-Paymaster of the Forces. During the ministry of Lord Chatham and the Duke of Grafton he supported the position his cousin Charles Townshend was in with regard to the American revenue program. Townshend was forced out of office in June 1768 by Grafton who wanted Rigby as Paymaster of the Forces to gain favour with the Duke of Bedford.