Brocket Hall early held variously of Hatfield Manor and the Bishop of Ely. Sir Matthew Lamb's son was Peniston Lamb, 1st Viscount Melbourne and he was often visited at Brocket Hall by the Prince Regent, who had a liaison with Lady Melbourne. The next owner was William Lamb, 2nd Viscount Melbourne, who was Queen Victoria's first Prime Minister from 1835–41, when she regularly visited. His wife, Lady Caroline, infamously had an affair with Lord Byron causing Lord Melbourne much embarrassment. For one of his birthdays she held a state banquet in the Saloon, at which she had herself served from a large
Elizabeth Lamb, Viscountess Melbourne Besford in the Duchess' novel "The Sylph". By contrast, she and the Duchess' sister, Henrietta Ponsonby, Countess of Bessborough, disliked each other intensely, and Lady Melbourne's enmity later extended to Henrietta's daughter, Lady Caroline Lamb. True to her practical nature, she did not allow this dislike to interfere with her support for her son William's marriage to Caroline, which was a social step up for the Lamb family, who were still, by the strict social standards of the age, considered to be newcomers to polite society. By the late 1770s, Lady Melbourne was romantically involved with George Wyndham, 3rd Earl