Brocket Hall early held variously of Hatfield Manor and the Bishop of Ely. Sir Matthew Lamb's son was Peniston Lamb, 1st Viscount Melbourne and he was often visited at Brocket Hall by the Prince Regent, who had a liaison with Lady Melbourne. The next owner was William Lamb, 2nd Viscount Melbourne, who was Queen Victoria's first Prime Minister from 1835–41, when she regularly visited. His wife, Lady Caroline, infamously had an affair with Lord Byron causing Lord Melbourne much embarrassment. For one of his birthdays she held a state banquet in the Saloon, at which she had herself served from a large
Peniston Lamb, 1st Viscount Melbourne of Cavan, but as it was an Irish peerage he was allowed to remain in the House of Commons. He was returned unopposed again as MP for Ludgershall at the elections in 1774 and 1780. In 1781 he was created Viscount Melbourne, of Kilmore in the County of Cavan, also in the Peerage of Ireland. He was appointed Gentleman of the Bedchamber to the Prince of Wales in 1783 and held the position until 1796. At the 1784 general election he stood for Malmesbury and was again returned unopposed. He switched again in 1790 and was returned unopposed at Newport,