Sackville was a notorious rake in his salad days. He was compelled to marry Lady Frances Leveson-Gower, daughter of John Leveson-Gower, 1st Earl Gower and sister of the Duchess of Bedford in 1744, after she gave birth to his child at Woburn. They eventually had issue two sons and one daughter. The couple's parents were furious, and the Prince of Wales compensated Sackville for any loss of income, making him a lord of his bedchamber in 1745, and thus securing him as a recruit to his party.