Brocket Hall early held variously of Hatfield Manor and the Bishop of Ely. Sir Matthew Lamb's son was Peniston Lamb, 1st Viscount Melbourne and he was often visited at Brocket Hall by the Prince Regent, who had a liaison with Lady Melbourne. The next owner was William Lamb, 2nd Viscount Melbourne, who was Queen Victoria's first Prime Minister from 1835–41, when she regularly visited. His wife, Lady Caroline, infamously had an affair with Lord Byron causing Lord Melbourne much embarrassment. For one of his birthdays she held a state banquet in the Saloon, at which she had herself served from a large
William Brett, 1st Viscount Esher that her father was one Louis Mayer. They had two sons, Reginald, and Eugène, and a daughter Violet, wife of William Humble Dudley Ward and mother of William Dudley Ward. Lord Esher died in London in May 1899, aged eighty-one, and was succeeded by his eldest son, Reginald. William Brett, 1st Viscount Esher William Baliol Brett, 1st Viscount Esher, PC (13 August 1815 – 24 May 1899), known as Sir William Brett between 1868 and 1883, was a British lawyer, judge, and Conservative politician. He was briefly Solicitor-General under Benjamin Disraeli and then served as a justice of the Court