Leveson Gower was born in Titsey Place near Oxted in Surrey, the seventh of twelve sons of Granville William Gresham Leveson-Gower JP DL FSA, by his wife The Hon Sophia Leveson Gower LJStJ (née Leigh). His father was a great-great grandson of John Leveson-Gower, 1st Earl Gower (descending from his youngest son, John), and served as Liberal MP for two years for Reigate from 1863 to 1865. His mother was the daughter of Chandos Leigh, 1st Baron Leigh and sister of Sir Edward Chandos Leigh QC and James Wentworth Leigh. His uncles, and brothers Frederick Leveson-Gower and Evelyn Marmaduke Gresham Leveson-Gower, also played first-class cricket.