Brocket Hall early held variously of Hatfield Manor and the Bishop of Ely. Sir Matthew Lamb's son was Peniston Lamb, 1st Viscount Melbourne and he was often visited at Brocket Hall by the Prince Regent, who had a liaison with Lady Melbourne. The next owner was William Lamb, 2nd Viscount Melbourne, who was Queen Victoria's first Prime Minister from 1835–41, when she regularly visited. His wife, Lady Caroline, infamously had an affair with Lord Byron causing Lord Melbourne much embarrassment. For one of his birthdays she held a state banquet in the Saloon, at which she had herself served from a large
William Lamb, 2nd Viscount Melbourne last time a British monarch attempted to appoint a government against a parliamentary majority. The next year, Melbourne was once again involved in a sex scandal. This time he was the victim of attempted blackmail from the husband of a close friend, society beauty and author Caroline Norton. The husband demanded £1,400, and when he was turned down he accused Melbourne of having an affair with his wife. At this time such a scandal would be enough to derail a major politician, so it is a measure of the respect contemporaries had for his integrity that Melbourne's government did not