Brocket Hall early held variously of Hatfield Manor and the Bishop of Ely. Sir Matthew Lamb's son was Peniston Lamb, 1st Viscount Melbourne and he was often visited at Brocket Hall by the Prince Regent, who had a liaison with Lady Melbourne. The next owner was William Lamb, 2nd Viscount Melbourne, who was Queen Victoria's first Prime Minister from 1835–41, when she regularly visited. His wife, Lady Caroline, infamously had an affair with Lord Byron causing Lord Melbourne much embarrassment. For one of his birthdays she held a state banquet in the Saloon, at which she had herself served from a large
Second Melbourne ministry in 1839, she refused to let Sir Robert Peel replace them with Conservative ladies. This was known as the Bedchamber Crisis, and led to Peel's refusal to form a government. Melbourne therefore resumed, and continued in office until the Conservatives finally won a House of Commons majority in the General Election of 1841. He was succeeded by Sir Robert Peel's second government. The 1841 votes of no confidence against the government of Viscount Melbourne were votes of no confidence in the government of William Lamb, 2nd Viscount Melbourne which occurred on the 7 June 1841. Melbourne lost the vote by