Brocket Hall early held variously of Hatfield Manor and the Bishop of Ely. Sir Matthew Lamb's son was Peniston Lamb, 1st Viscount Melbourne and he was often visited at Brocket Hall by the Prince Regent, who had a liaison with Lady Melbourne. The next owner was William Lamb, 2nd Viscount Melbourne, who was Queen Victoria's first Prime Minister from 1835–41, when she regularly visited. His wife, Lady Caroline, infamously had an affair with Lord Byron causing Lord Melbourne much embarrassment. For one of his birthdays she held a state banquet in the Saloon, at which she had herself served from a large
Peniston Lamb, 1st Viscount Melbourne many affairs. George is reputed to be the son of George IV, with William and Emily allegedly fathered by Lord Egremont. Whether Melbourne was made unhappy by his wife's affairs is unclear: he was a mild, easygoing and rather stupid man who avoided trouble, and invariably deferred to his wife, who was by far the stronger and more intelligent partner in the marriage. Their one serious quarrel was caused by the death of their eldest son Pen (who was undoubtedly Melbourne's child); he angrily refused to make the same allowance to William (who was almost certainly not Melbourne's child) as