Brocket Hall early held variously of Hatfield Manor and the Bishop of Ely. Sir Matthew Lamb's son was Peniston Lamb, 1st Viscount Melbourne and he was often visited at Brocket Hall by the Prince Regent, who had a liaison with Lady Melbourne. The next owner was William Lamb, 2nd Viscount Melbourne, who was Queen Victoria's first Prime Minister from 1835–41, when she regularly visited. His wife, Lady Caroline, infamously had an affair with Lord Byron causing Lord Melbourne much embarrassment. For one of his birthdays she held a state banquet in the Saloon, at which she had herself served from a large
William Lamb, 2nd Viscount Melbourne a lurid fashion, which caused William even greater embarrassment, while the spiteful caricatures of leading society figures made them several influential enemies. Eventually the two were reconciled, and, though they separated in 1825, her death in 1828 affected him considerably. In 1816, Lamb was returned for Peterborough by Whig grandee Lord Fitzwilliam. He told Lord Holland that he was committed to the Whig principles of the Glorious Revolution but not to "a heap of modern additions, interpolations, facts and fictions". He therefore spoke against parliamentary reform, and voted for the suspension of "habeas corpus" in 1817 when sedition was rife.