Buckingham was the second child and only son of Richard, 2nd Duke of Buckingham and Chandos, and his wife Lady Mary, younger daughter of John Campbell, 4th Earl of Breadalbane and Holland (later The 1st Marquess of Breadalbane). His sister, Lady Anna Eliza Mary Gore-Langton, was a women's rights campaigner. As his father's son and heir apparent, he was styled Earl Temple from birth. He was 15 years of age when his paternal grandfather died and his father became the 2nd Duke of Buckingham and Chandos. From this time, he was styled Marquess of Chandos until he succeeded his father as Duke.