His wife, Lady Caroline Lamb, infamously had an affair with Lord Byron, causing Lord Melbourne much embarrassment.
Lady Caroline Lamb's relationship with Lord Byron had a significant impact on Lord Melbourne as it resulted in a public scandal that caused him embarrassment.