On 18 October 1744 Carteret became 2nd Earl Granville on the death of his mother. His first wife Lady Frances Worsley died on 20 June 1743 at Hanover, and in April 1744 he married Lady Sophia Fermor, a daughter of Thomas Fermor, 1st Earl of Pomfret, a fashionable beauty and "reigning toast" of London society, who was younger than his daughters. Granville's ostentatious performance of the part of lover was ridiculed by Horace Walpole as "The nuptials of our great Quixote and the fair Sophia" and "My lord stayed with her there till four in the morning. They are all fondnesswalk together, and stop every five steps to kiss."