Brocket Hall early held variously of Hatfield Manor and the Bishop of Ely. Sir Matthew Lamb's son was Peniston Lamb, 1st Viscount Melbourne and he was often visited at Brocket Hall by the Prince Regent, who had a liaison with Lady Melbourne. The next owner was William Lamb, 2nd Viscount Melbourne, who was Queen Victoria's first Prime Minister from 1835–41, when she regularly visited. His wife, Lady Caroline, infamously had an affair with Lord Byron causing Lord Melbourne much embarrassment. For one of his birthdays she held a state banquet in the Saloon, at which she had herself served from a large
Frederick Lamb, 3rd Viscount Melbourne Baron Beauvale, of Beauvale in the County of Nottingham. In 1848 he succeeded his elder brother as third Viscount Melbourne. Despite a certain personal distance between them, Lord Palmerston, as Foreign Secretary placed great confidence in Lamb, wrote to him in a courteous style very different from his usual brusque manner, and left the running of the Vienna Embassy almost entirely in his hands. The coolness was due to Palmerston's decades-long affair with Lamb's sister Emily, Lady Cowper; Lamb disapproved of the affair and disapproved equally of their eventual marriage, although this proved to be very happy. Palmerston's biographer notes