The principal opposition to Pitt was a relatively weak faction of Whigs, led by Charles James Fox.
The significant resistance against Pitt came from a diminutive contingent of Whigs, guided by Charles James Fox.