Brocket Hall early held variously of Hatfield Manor and the Bishop of Ely. Sir Matthew Lamb's son was Peniston Lamb, 1st Viscount Melbourne and he was often visited at Brocket Hall by the Prince Regent, who had a liaison with Lady Melbourne. The next owner was William Lamb, 2nd Viscount Melbourne, who was Queen Victoria's first Prime Minister from 1835–41, when she regularly visited. His wife, Lady Caroline, infamously had an affair with Lord Byron causing Lord Melbourne much embarrassment. For one of his birthdays she held a state banquet in the Saloon, at which she had herself served from a large
William Lamb, 2nd Viscount Melbourne residence, Petworth, Lamb was a visitor until the Earl's death; he was called to his bedside when he was dying. Lamb nevertheless stated that Egremont being his father was 'all a lie'. He was educated at Eton, Trinity College, Cambridge and the University of Glasgow. Against the background of the Napoleonic Wars, Lamb served at home as captain (1803) and major (1804) in the Hertfordshire Volunteer Infantry. He succeeded his elder brother as heir to his father's title in 1805, and married Lady Caroline Ponsonby, an Anglo-Irish aristocrat. The following year, he was elected to the British House of Commons