Hervey had been at one time on very friendly terms with Frederick, Prince of Wales, but in about 1732 they quarrelled, apparently because they were rivals for the affection of Anne Vane. These differences probably account for the scathing picture he draws of the Prince's callous conduct. Hervey had been hesitating between William Pulteney (afterwards earl of Bath) and Robert Walpole, but in 1730 he definitely took sides with Walpole, of whom he was thenceforward a faithful adherent. He was assumed by Pulteney to be the author of "Sedition and Defamation display'd, with a Dedication to the patrons of The Craftsman" (1731). Pulteney, who, up to this time, had been a firm friend of Hervey, replied with "A Proper Reply to a late Scurrilous Libel", and the quarrel resulted in a duel from which Hervey narrowly escaped with his life.