Brocket Hall early held variously of Hatfield Manor and the Bishop of Ely. Sir Matthew Lamb's son was Peniston Lamb, 1st Viscount Melbourne and he was often visited at Brocket Hall by the Prince Regent, who had a liaison with Lady Melbourne. The next owner was William Lamb, 2nd Viscount Melbourne, who was Queen Victoria's first Prime Minister from 1835–41, when she regularly visited. His wife, Lady Caroline, infamously had an affair with Lord Byron causing Lord Melbourne much embarrassment. For one of his birthdays she held a state banquet in the Saloon, at which she had herself served from a large
Frederick Lamb, 3rd Viscount Melbourne that the marriage coincided with the early stages of the Oriental Crisis of 1840, and that the two men, although they were then personally barely on speaking terms, co-operated in an entirely professional way to resolve it. Palmerston, in addition to his real respect for Lamb, was anxious not to quarrel with him for Emily's sake: as Charles Greville remarked: "the Chief (Palmerston) is devoted to the sister and the sister to the brother". Relations between the two men became friendlier in later years, partly because both Palmerston and Emily were fond of Fred's wife Alexandrina. Lord Melbourne died childless