Brocket Hall early held variously of Hatfield Manor and the Bishop of Ely. Sir Matthew Lamb's son was Peniston Lamb, 1st Viscount Melbourne and he was often visited at Brocket Hall by the Prince Regent, who had a liaison with Lady Melbourne. The next owner was William Lamb, 2nd Viscount Melbourne, who was Queen Victoria's first Prime Minister from 1835–41, when she regularly visited. His wife, Lady Caroline, infamously had an affair with Lord Byron causing Lord Melbourne much embarrassment. For one of his birthdays she held a state banquet in the Saloon, at which she had herself served from a large
William Lamb, 2nd Viscount Melbourne as the Whig MP for Leominster. For the election in 1806 he moved to the seat of Haddington Burghs, and for the 1807 election he successfully stood for Portarlington (a seat he held until 1812). Lamb first came to general notice for reasons he would rather have avoided: his wife had a public affair with Lord Byron—she coined the famous characterisation of Byron as "mad, bad and dangerous to know". The resulting scandal was the talk of Britain in 1812. Lady Caroline published a Gothic novel, "Glenarvon", in 1816; this portrayed both the marriage and her affair with Byron in