Brocket Hall early held variously of Hatfield Manor and the Bishop of Ely. Sir Matthew Lamb's son was Peniston Lamb, 1st Viscount Melbourne and he was often visited at Brocket Hall by the Prince Regent, who had a liaison with Lady Melbourne. The next owner was William Lamb, 2nd Viscount Melbourne, who was Queen Victoria's first Prime Minister from 1835–41, when she regularly visited. His wife, Lady Caroline, infamously had an affair with Lord Byron causing Lord Melbourne much embarrassment. For one of his birthdays she held a state banquet in the Saloon, at which she had herself served from a large
Frederick Lamb, 3rd Viscount Melbourne marriage: even though Alexandrina was more than thirty years her husband's junior, he was described as being "as handsome and debonair at sixty as he had been at twenty-five." William, Frederick and their sister Emily Lamb, Countess Cowper remained close all their lives, although Frederick and Emily disliked William's wife Lady Caroline Lamb, whom they called "the little beast". He served as British Ambassador to Vienna ending in 1841. He was invested as a Knight Grand Cross of the Order of the Bath and admitted to the Privy Council in 1822. In 1839 he was raised to the peerage as