Brocket Hall early held variously of Hatfield Manor and the Bishop of Ely. Sir Matthew Lamb's son was Peniston Lamb, 1st Viscount Melbourne and he was often visited at Brocket Hall by the Prince Regent, who had a liaison with Lady Melbourne. The next owner was William Lamb, 2nd Viscount Melbourne, who was Queen Victoria's first Prime Minister from 1835–41, when she regularly visited. His wife, Lady Caroline, infamously had an affair with Lord Byron causing Lord Melbourne much embarrassment. For one of his birthdays she held a state banquet in the Saloon, at which she had herself served from a large
Elizabeth Lamb, Viscountess Melbourne Sir Peniston Lamb, 2nd Baronet; they married in London on 13 April 1769. The couple lived at Melbourne House in Piccadilly and Elizabeth quickly became a well-known figure in London Whig society. She was apparently untroubled by her husband's infidelity early in their marriage with the actress and courtesan Sophia Baddeley, just as he came to tolerate her numerous love affairs. Peniston was raised to the Peerage of Ireland as Lord Melbourne, Baron of Kilmore, in 1770 and Viscount Melbourne in 1781. As well as Melbourne House, the family had country residences at Brocket Hall in Hertfordshire and Melbourne Hall