Brummell was born in London, the younger son of Jane (née Richardson, daughter of the Keeper of the Lottery Office) and William Brummell (d. 1794), Private Secretary to the Prime Minister, Lord North. On his retirement from politics, William had bought Donnington Grove in Berkshire, and served as High Sheriff of Berkshire in 1788. William was the son of another William Brummell (d. 1770), who had been valet to a Lincolnshire politician, Charles Monson, and, reckoned "an excellent servant", met with some success despite his modest origins through patronage and good fortune. He was in business as a confectioner in Bury Street, "in an area notorious for... high-class brothels", letting some rooms in the family's house for boarding. The statesman Charles Jenkinson, 1st Earl of Liverpool stayed there for a time, and got the younger William a clerical position at the Treasury, which led to his successful career.