In 1802 he returned to Europe, and the next year married Lady Charlotte Cadogan, by whom he had three sons and a daughter before she abandoned him in 1809, running off with Lord Paget, a talented cavalry officer. His wife divorced him in Scotland in 1810. Although, at first, Paget's career was badly affected by the affair, as he was later unable to serve under Wellesley's brother Wellington in the Peninsular Campaign due to the bad blood, Paget later distinguished himself under Wellington at Waterloo.