In 1778 he eloped with Amelia Osborne, Marchioness of Carmarthen, daughter ofRobert Darcy, 4th Earl of Holderness, to Europe and they married after she obtained a divorce from her husband the Marquess of Carmarthen, who in 1789 became Francis Osborne, 5th Duke of Leeds. They married on 1 June 1779 at London, England, and had a daughter, Augusta Maria Byron. Amelia Osborne died in 1784.