He married Mary Crichton-Stuart, daughter of John Stuart, 3rd Earl of Bute, and Mary Wortley-Montagu, 1st Baroness Mount Stuart, on 7 September 1761 and had a string of mistresses. He fell in love with the daughter of one of his tenants and made her his mistress keeping her in luxury. When she died he could not endure to have her buried and the body remained lying in bed until the increasing putrefaction became unbearable. He then had her body placed in a glass topped coffin that was placed in a cupboard. Eventually her body was buried in Paddington cemetery.