Brocket Hall early held variously of Hatfield Manor and the Bishop of Ely. Sir Matthew Lamb's son was Peniston Lamb, 1st Viscount Melbourne and he was often visited at Brocket Hall by the Prince Regent, who had a liaison with Lady Melbourne. The next owner was William Lamb, 2nd Viscount Melbourne, who was Queen Victoria's first Prime Minister from 1835–41, when she regularly visited. His wife, Lady Caroline, infamously had an affair with Lord Byron causing Lord Melbourne much embarrassment. For one of his birthdays she held a state banquet in the Saloon, at which she had herself served from a large
Elizabeth Lamb, Viscountess Melbourne boy's godfather. Later in life, Lady Melbourne formed a friendship with the poet Lord Byron. She became his confidante during his affair with her daughter-in-law Lady Caroline Lamb. Although she approved of her son's marriage on social grounds, Lady Melbourne disliked Caroline intensely and their relationship was always bad. By contrast she liked Byron, and blamed him neither for having the affair with Caroline, nor for ending it. Byron later married Lady Melbourne's niece, Anne Isabella Milbanke. Lord Melbourne was made a peer of the United Kingdom as Baron Melbourne of Melbourne in 1815. Lady Melbourne died on 6 April