Hervey was bisexual. He had an affair with Anne Vane, and possibly with Lady Mary Wortley Montagu and Princess Caroline. He lived with Stephen Fox often during the decade after he followed him to Italy in 1728. He wrote passionate love letters to Francesco Algarotti, whom he first met in 1736. He may have had a sexual affair with Prince Frederick before their friendship dissolved. He was in fact denounced as a sexually ambiguous figure in his time most notably by William Pulteney, then leader of the Opposition and as cited above, by Alexander Pope in his "Sporus" portrait: "Let Sporus tremble/What that thing of silk...His wit all seesaw between that and this/Now high, now low, now master up, now miss/And he himself one vile antithesis...". He was also attracted to Henry Fox before his affair with Stephen Fox.