Under the Chatham Ministry, Granby was appointed commander-in-chief on 13 August 1766. Despite rumours of his retirement, he vigorously electioneered during the 1768 season, and increased the Rutland interests seats to seven, at some expense. With the resignation of Chatham, he found himself somewhat isolated in the Grafton Ministry. While he had opposed the attempts of the government to expel Wilkes from his seat in Middlesex, his personal dislike of Wilkes overcame his principles, and he voted in favor of the expulsion on 3 February 1769 and for the seating of Henry Luttrell afterwards. It was to prove a serious political mistake. Junius, a political writer, attacked the ministry accusing Granby of servility towards the court and personal corruption. Granby's great popularity might have let him ride out the affair, but his reversal on Wilkes provided new ammunition. Worse still, a reply to Junius by his friend Sir William Draper, intended in his defence, essentially validated the charge that the hard-drinking and personable Granby was easily imposed upon by less scrupulous acquaintances.